How Does RLHF Work? (Reward Modeling, PPO & Direct Preference Optimization / DPO)

The Alignment Problem: From Pre-Training to Supervised Fine-Tuning

Pre-trained foundation models are autoregressive next-token predictors trained on internet-scale text to minimize cross-entropy loss [1,2]. While pre-training endows the base model $\pi_{\text{base}}$ with extensive world knowledge and syntactic capabilities, it does not inherently align the model to follow instructions, avoid toxic completions, or provide concise, helpful answers [1,3]. An autoregressive completion of a question might simply hallucinate a follow-up question or generate malicious code found in training web corpora [1,2].

The first stage of alignment is Supervised Fine-Tuning (SFT), where the base model is trained on curated demonstration datasets comprising high-quality prompt-response pairs $(x, y)$ written or vetted by human annotators [1,4]. SFT shifts the base model distribution toward conversational helpfulness, producing an initial policy $\pi^{\text{SFT}}(y|x)$ [1,3]. However, supervised learning alone suffers from distributional drift (exposure bias) and cannot efficiently scale to capture subtle human preferences across ambiguous open-ended outputs [1,4].

"Autoregressive pre-training optimizes for raw probability of internet text, whereas alignment optimizes for human intent, truthfulness, and safety."

Reward Modeling via the Bradley-Terry Preference Formulation

To overcome the bottlenecks of manual demonstration writing, researchers collect paired model completions $(y_w, y_l)$ for a given prompt $x$, where human labelers indicate that completion $y_w$ is preferred over $y_l$ ($y_w \succ y_l$) [1,3,5]. A separate neural network, the Reward Model $r_\theta(x, y)$, is trained to output a scalar score representing human preference [1,3].

Reward model training formalizes human preferences using the Bradley-Terry pairwise comparison model [1,5]. The probability that a human annotator prefers $y_w$ over $y_l$ is modeled as the logistic sigmoid of their score difference:

$$P(y_w \succ y_l | x) = \sigma\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right) = \frac{1}{1 + e^{-\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right)}}$$

The reward model parameters $\theta$ are optimized by minimizing the negative log-likelihood loss over the dataset of comparisons $\mathcal{D} = \{(x, y_w, y_l)\}$: $\mathcal{L}_R(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\left[\log \sigma\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right)\right]$ [1,3,5].

Policy Optimization via PPO and the KL Penalty

In standard RLHF, the fine-tuned language model acts as an agent interacting with an environment where the prompt $x$ is the state, the generated token sequence $y$ is the action, and the reward model provides the terminal scalar reward $r_\theta(x, y)$ [1,3,4]. Optimization is executed using Proximal Policy Optimization (PPO), a clipped actor-critic policy gradient algorithm [1,6].

Without constraints, the policy $\pi_\phi$ would exploit flaws in the reward model—a failure mode known as reward hacking or "Goodhart's Law in AI"—producing degenerate, repetitive, or unreadable strings that score arbitrarily high rewards [1,3,6]. To prevent this, the RL objective incorporates a Kullback-Leibler (KL) divergence penalty against the reference supervised policy $\pi^{\text{SFT}}$:

$$\max_{\phi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\phi(y|x)}\left[ r_\theta(x, y) - \beta \, D_{\text{KL}}\left(\pi_\phi(y|x) \parallel \pi^{\text{SFT}}(y|x)\right) \right]$$

The hyperparameter $\beta$ governs the regularization strength, ensuring the policy does not drift into nonsensical regions of token space while maximizing human alignment [1,3,6].

"The KL-divergence penalty acts as an anchor, preventing the generative policy from exploiting flaws in the reward model."

Direct Preference Optimization (DPO): Bypassing the Reinforcement Learning Loop

While PPO-based RLHF proved foundational for models like InstructGPT and Claude, it is notoriously unstable, computationally expensive, and requires maintaining four large models simultaneously in GPU memory: the actor $\pi_\phi$, reference policy $\pi^{\text{ref}}$, reward model $r_\theta$, and value network $V_\psi$ [1,3,7].

In 2023, Rafael Rafailov et al. derived Direct Preference Optimization (DPO), mathematically proving that the constrained RL objective has an exact closed-form analytical solution for the optimal reward in terms of policy probabilities [7]:

$$r(x, y) = \beta \log \frac{\pi^*(y|x)}{\pi^{\text{ref}}(y|x)} + \beta \log Z(x)$$

By substituting this analytical reward directly into the Bradley-Terry loss, DPO optimizes the language model policy $\pi_\theta$ directly on preference data using simple binary cross-entropy loss without training a separate reward model or sampling actions during training:

$$\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi^{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi^{\text{ref}}(y_l|x)}\right)\right]$$

DPO has largely democratized post-training alignment, delivering stable, deterministic convergence across open-source and proprietary foundation models [7].