RL for alignment: RLHF and its successors

Why this module

A pretrained model predicts the next token of internet text. That is not the same as being helpful, honest or safe, and for a driving VLA it is not the same as driving well. Alignment here means the practical step of fine-tuning a capable model so that its outputs match what people actually want. Most of today's recipe is reinforcement learning, so this module assumes Reinforcement learning from scratch (states, rewards, returns, policy gradients, baselines).

1. Why not just more supervised fine-tuning?

Supervised fine-tuning (SFT) on demonstrations works and is always the first step. Its limits:

• Demonstrations are expensive and capped by the demonstrator. Writing the ideal answer is hard; you cannot demonstrate behaviour better than your own.
• Imitation cannot learn "better than". It only sees good examples, never which of two outputs is worse and why.
• Judging is easier than doing. Most people can say which of two summaries is better, or which of two trajectories is more comfortable, faster than they can write or drive the ideal one.

That last point is the foundation of learning from human feedback. "Deep reinforcement learning from human preferences" (1706.03741) trained Atari and simulated robot locomotion agents from non-expert humans comparing pairs of short clips, with feedback on less than one percent of the agent's interactions and about an hour of human time for new behaviours. Applied to language, "Learning to summarize from human feedback" (2009.01325) reported that summaries from optimising a learned human-preference reward were preferred by humans over those from optimising ROUGE, and outperformed the human reference summaries on the Reddit TL;DR dataset.

2. The RLHF recipe

InstructGPT (2203.02155) set the three-step recipe most systems still follow:

1. SFT: fine-tune on human-written demonstrations.
2. Reward model: humans rank several outputs for the same prompt; train a model to predict which one they prefer.
3. RL: optimise the SFT model against the reward model with PPO, while penalising drift from the SFT model.

In the paper's human evaluations, outputs of the 1.3-billion-parameter InstructGPT were preferred to those of the 175-billion-parameter GPT-3, a model about 100 times larger.

RLHF as an RL problem

The reward arrives only at the end of the response: a credit-assignment problem across hundreds of tokens. This is where the value network (critic) of PPO, or the group baseline of GRPO, earns its keep.

3. Step 2: a reward model from comparisons

People are bad at giving absolute scores ("is this answer a 6 or a 7?") and good at comparisons. The Bradley–Terry model links a scalar reward $r$ to comparisons: the probability that response $y_w$ is preferred to $y_l$ is

where $\sigma$ is the logistic sigmoid. Training the reward model is then ordinary binary classification: minimise $-\log \sigma(r(y_w) - r(y_l))$ over the labelled pairs. Only differences in reward matter, so the reward's scale and offset are arbitrary.

def bradley_terry_loss(chosen: torch.Tensor, rejected: torch.Tensor) -> torch.Tensor:
    """Reward-model loss on preference pairs: -log sigmoid(r_chosen - r_rejected).

    The Bradley-Terry model says P(chosen preferred) = sigmoid(r_chosen - r_rejected),
    so this is the negative log-likelihood of the observed human choices.
    """
    return -torch.nn.functional.logsigmoid(chosen - rejected).mean()


def collect_preferences(
    features: torch.Tensor, pairs: int = 2000, seed: int = 0
) -> tuple[torch.Tensor, torch.Tensor]:
    """Simulated labellers compare pairs of responses sampled from the reference model.

    Each label is noisy: the better response (by ``true_reward``) wins with
    probability sigmoid(difference in true reward).

    Returns:
        Indices of the chosen and rejected responses, each shape (pairs,).
    """
    generator = torch.Generator().manual_seed(seed)
    probs = reference_logp(features).exp()
    a = torch.multinomial(probs, pairs, replacement=True, generator=generator)
    b = torch.multinomial(probs, pairs, replacement=True, generator=generator)
    rewards = true_reward(features)
    a_wins = torch.rand(pairs, generator=generator) < torch.sigmoid(
        rewards[a] - rewards[b]
    )
    return torch.where(a_wins, a, b), torch.where(a_wins, b, a)


def fit_reward_model(
    features: torch.Tensor,
    chosen: torch.Tensor,
    rejected: torch.Tensor,
    steps: int = 500,
) -> torch.Tensor:
    """Fits a linear reward model r(x) = w . x to the preferences; returns w."""
    weights = torch.zeros(features.shape[1], requires_grad=True)
    optimizer = torch.optim.Adam([weights], lr=0.05)
    for _ in range(steps):
        rewards = features @ weights
        loss = bradley_terry_loss(rewards[chosen], rewards[rejected])
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return weights.detach()

The toy in the reference code: the "model" picks one of 2000 candidate responses, each with a quality and a length. Simulated labellers prefer quality, and prefer longer answers only up to a length of 3. Pairs are sampled from the SFT model, which mostly writes short answers, and each label is noisy (the better response wins with probability given by the formula above). The fitted linear reward model weights quality at 0.98 and length at 0.23: in the data it saw, longer really was better. Keep that in mind for section 5.

4. Step 3: optimise, but stay close

The RL step maximises reward minus a penalty for moving away from the SFT model $\pi_{\text{ref}}$:

The KL term does two jobs: it keeps the model fluent (the SFT model already writes well), and it limits how far the policy can wander into regions where the reward model has never been checked. $\beta$ sets the trade-off.

This objective has an exact solution:

Start from the SFT model and up-weight each response exponentially in its reward; a small $\beta$ up-weights aggressively. For a real language model this cannot be computed directly (there are too many possible responses to normalise over), so PPO approaches it by gradient steps: sample responses, score them, compute advantages, take a clipped step. In practice that means four models in memory: the policy, the frozen reference, the reward model and PPO's value model. GRPO drops the value model by using a group of samples as the baseline (the RL for policies and reasoning module).

def kl_regularised_policy(
    ref_logp: torch.Tensor, reward: torch.Tensor, beta: float
) -> torch.Tensor:
    """The policy that maximises E[reward] - beta * KL(policy || reference).

    The optimum has a closed form: pi(y) is proportional to pi_ref(y) exp(r(y) / beta).
    A small beta trusts the reward and moves far from the reference; a large beta
    stays close to it. PPO in RLHF approximates this policy by gradient steps; DPO
    starts from this formula.

    Raises:
        ValueError: If ``beta`` is not positive.
    """
    if beta <= 0:
        raise ValueError(f"beta must be positive, got {beta}")
    return torch.log_softmax(ref_logp + reward / beta, dim=0)


def kl_divergence(logp: torch.Tensor, ref_logp: torch.Tensor) -> float:
    """KL(policy || reference) for two log-probability vectors over the same items."""
    return (logp.exp() * (logp - ref_logp)).sum().item()

A test checks that plain gradient ascent on the objective converges to this closed form. The formula is also the starting point of DPO (section 6).

5. Goodhart's law: over-optimising the reward model

The reward model is a proxy for what people want, fitted on limited data. Push hard enough and the policy finds responses the proxy scores highly but people would not. "Scaling Laws for Reward Model Overoptimization" (2210.10760) studies this with a fixed "gold" reward model standing in for humans: as optimisation pressure grows, the gold score follows different curves for RL and best-of-n sampling, with coefficients that scale smoothly with reward model size, and the paper also studies the effect of data size and the KL coefficient.

The toy shows the same shape. Lowering $\beta$ lets the policy move further from the SFT model; the learned reward keeps rising, but the true reward peaks and then falls, because the policy discovers that the reward model's "longer is better" never stops:

(3 seeds; uv run --group tutorial python scripts/tutorial_results.py rlhf.) Lessons that carry over:

• Watch a held-out signal, not the training reward. The learned reward said "keep going" the whole way. Only an independent evaluation shows the turn.
• The KL penalty is a safety margin, not a nuisance. It is the knob that kept the policy where the reward model is trustworthy.
• More and better-targeted preference data moves the peak. The toy's reward model was wrong only because the data never showed long answers; collecting comparisons on the policy's new outputs is why practical RLHF repeats data collection.

6. Variants you will meet

DPO: no reward model, no RL loop

Rearranging the closed-form optimum of section 4 gives the reward in terms of the policy: $r(y) = \beta \log \frac{\pi(y)}{\pi_{\text{ref}}(y)} + \text{const}$. Substituting into the Bradley–Terry loss gives a loss on the policy directly, trained on the same comparison pairs. That is Direct Preference Optimization, with code and a toy in Preference learning and offline RL.

AI feedback instead of human feedback

Human labels are slow and costly. Constitutional AI (2212.08073) replaces human harmfulness labels with a written list of principles (a "constitution"): a supervised phase has the model critique and revise its own responses, and an RL phase trains a preference model on AI-generated comparisons, which the authors call RL from AI Feedback (RLAIF). "RLAIF vs. RLHF" (2309.00267) reports RLAIF performing comparably to RLHF on summarisation and helpful and harmless dialogue, and that getting rewards directly from an LLM during RL, without training a reward model, did better still.

Verifiable rewards

When an answer can be checked, no learned reward model is needed: a maths answer matches or not, code passes its tests or not. Tulu 3 (2411.15124) names this Reinforcement Learning with Verifiable Rewards (RLVR). DeepSeek-R1 (2501.12948) used rule-based rewards with GRPO to train long reasoning (the Reasoning module). A checker cannot be flattered, which removes most of section 5's problem, but it only covers what can be checked.

Outcome versus process rewards

Rewarding only the final answer gives one signal for a long chain of reasoning; rewarding each step gives denser credit. "Let's Verify Step by Step" (2305.20050) reports process supervision beating outcome supervision on the MATH problems it studies.

7. Alignment for a driving VLA

Alpamayo-R1's RL stage is this module applied to driving (the full table is in RL for policies and reasoning):

and it optimises with GRPO. Each kind brings its own failure: the LLM judge can be flattered, the rule check can be gamed by trivial reasoning, and the hand-written reward can be hacked (standing still never collides). Comparisons are a natural fit for what is hard to write down in driving, such as comfort, assertiveness or politeness: "which of these two trajectories would you rather ride in?" is easy to answer.

Reference implementation (PyTorch)

libs/tutorial/rlhf.py; tests are in libs/tutorial/tests/test_rlhf.py: Bradley–Terry loss values, simulated labels mostly favour the truly better response, the reward model learns that quality and (in its data) length help, the closed-form KL-regularised policy equals gradient ascent on the objective, beta controls distance from the reference, and for three seeds the learned reward keeps rising while the true reward peaks and falls. Run uv run --group tutorial python -m libs.tutorial.rlhf.

Check yourself

1. Give two reasons to collect comparisons rather than demonstrations.
2. In RLHF for a language model, what are the state, the action and the episode?
3. What does the Bradley–Terry model assume, and why does only the difference in reward matter?
4. What are the two jobs of the KL penalty?
5. In the toy, why does the true reward fall while the learned reward keeps rising?
6. What does a verifiable reward remove, and what can it not cover?

Quiz

Answer each question; feedback appears as soon as you choose.

Sources

• Christiano et al., Deep reinforcement learning from human preferences, 2017.
• Stiennon et al., Learning to summarize from human feedback, 2020.
• Ouyang et al., Training language models to follow instructions with human feedback, 2022 (InstructGPT).
• Gao, Schulman and Hilton, Scaling Laws for Reward Model Overoptimization, 2022.
• Bai et al., Constitutional AI: Harmlessness from AI Feedback, 2022.
• Lee et al., RLAIF vs. RLHF, 2023.
• Rafailov et al., Direct Preference Optimization, 2023.
• Lambert et al., Tulu 3, 2024 (RLVR).
• DeepSeek-AI, DeepSeek-R1, 2025.
• Lightman et al., Let's Verify Step by Step, 2023.
• Alpamayo-R1, arXiv 2511.00088, Sec. 5.3.