Preference learning and offline RL

Two ways to learn without a hand-written reward

The Reinforcement learning module assumed a reward function. Often you do not have one, but you have either human preferences between two outputs, or a large log of past behaviour you cannot afford to re-collect. This module covers one method for each.

Preferences: RLHF and DPO

Training language models to follow instructions with human feedback (2203.02155) fits a reward model to human comparisons and then uses RL against it (RL for alignment walks through that recipe, the KL-regularised objective and reward over-optimisation step by step). Direct Preference Optimization (2305.18290) shows the reward model and the RL loop can be skipped: a new parameterization lets the optimal policy be extracted in closed form, giving a simple classification-style loss on (chosen, rejected) pairs:

$\mathcal{L}_{\text{DPO}} = -\log \sigma\!\Big(\beta\big[(\log\pi_\theta(y_w) - \log\pi_\theta(y_l)) - (\log\pi_{\text{ref}}(y_w) - \log\pi_{\text{ref}}(y_l))\big]\Big)$

Read it as: raise the probability of the chosen response $y_w$ relative to the rejected one $y_l$, measured against a frozen reference model. The weight $\beta$ sets how far the policy may move from that reference.

In our toy (300 steps, a uniform reference; mean and standard deviation over 5 seeds from uv run --group tutorial python scripts/tutorial_results.py dpo), the highest-reward response got probability 1.00 at beta = 0.05 (KL to the reference 1.39), 0.93 ± 0.01 at beta = 0.5 (KL 1.07 ± 0.04), and 0.34 ± 0.01 at beta = 5 (KL 0.02), where the policy has barely left the reference.

Reference implementation (PyTorch)

Tests are in libs/tutorial/tests/test_dpo.py, including that the loss equals log 2 when the policy equals the reference and that a larger beta stays closer to it. Run uv run --group tutorial python -m libs.tutorial.dpo.

def dpo_loss(
    logp_chosen: torch.Tensor,
    logp_rejected: torch.Tensor,
    ref_logp_chosen: torch.Tensor,
    ref_logp_rejected: torch.Tensor,
    beta: float = 0.1,
) -> torch.Tensor:
    """The DPO loss for a batch of preference pairs.

    Args:
        logp_chosen: Log-probability of the preferred response under the policy.
        logp_rejected: Log-probability of the rejected response under the policy.
        ref_logp_chosen: The same under the frozen reference policy.
        ref_logp_rejected: The same under the frozen reference policy.
        beta: How strongly to stay near the reference; larger means closer.

    Returns:
        The mean of -log sigmoid(beta * (policy margin - reference margin)).
    """
    margin = (logp_chosen - logp_rejected) - (ref_logp_chosen - ref_logp_rejected)
    return -torch.nn.functional.logsigmoid(beta * margin).mean()

def train_dpo(beta: float, steps: int = 300, seed: int = 0) -> torch.Tensor:
    """Trains a categorical policy with DPO and returns its final logits.

    The reference policy is uniform, which is also the starting policy.
    """
    generator = torch.Generator().manual_seed(seed)
    logits = torch.zeros(NUM_RESPONSES, requires_grad=True)
    ref_logp = torch.log_softmax(torch.zeros(NUM_RESPONSES), dim=0)
    optimizer = torch.optim.Adam([logits], lr=0.05)
    for _ in range(steps):
        chosen, rejected = sample_pairs(64, generator)
        logp = torch.log_softmax(logits, dim=0)
        loss = dpo_loss(
            logp[chosen], logp[rejected], ref_logp[chosen], ref_logp[rejected], beta
        )
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return logits.detach()

The reward hiding inside the policy

DPO never fits a reward model, but its loss is the Bradley–Terry loss for the quantity $\hat r(y) = \beta \log \frac{\pi_\theta(y)}{\pi_{\text{ref}}(y)}$ (the RL for alignment module derives why). A trained policy therefore carries a reward ranking inside it:

def implicit_reward(logits: torch.Tensor, beta: float) -> torch.Tensor:
    """The reward DPO's policy implies: beta * log(pi / pi_ref) for a uniform pi_ref.

    DPO never fits a reward model, but its loss is the Bradley-Terry loss for exactly
    this quantity, so the trained policy carries a reward ranking inside it (up to an
    additive constant that cancels in every comparison).
    """
    return beta * (
        logits.log_softmax(0) + torch.log(torch.tensor(float(NUM_RESPONSES)))
    )


def win_rate() -> torch.Tensor:
    """How often each response is the chosen one when it appears in a random pair."""
    reward = HIDDEN_REWARD
    pairwise = torch.sigmoid(reward[:, None] - reward[None, :])  # P(row beats column)
    pairwise.fill_diagonal_(0.0)
    return pairwise.sum(dim=1) / (NUM_RESPONSES - 1)

On the toy at $\beta = 0.5$ the implied rewards are −1.75, −0.77, +0.66 and −1.51 for the four responses, whose hidden rewards are 0, 1, 2.5 and 0.5: a correlation of 0.99, recovered from comparisons alone (a test checks that the best response gets the highest implied reward).

A chosen response can lose probability

Response 1 has hidden reward 1.0, so in a random comparison it is the chosen one about half the time. Yet after DPO its probability is 0.05, down from 0.25. The loss does not say "make every chosen response likelier"; it says "raise the chosen one relative to the rejected one". When the mass can move to a response that beats everything, both the middling chosen and the rejected responses lose probability. In real models the same effect appears as a chosen answer's likelihood falling during DPO training, so watch the log-probability of chosen responses as well as the loss.

Where preferences matter for robots and driving: comfort, politeness and "which of these two trajectories is better" are easier to compare than to write as a reward.

Offline RL

Offline RL learns a policy from a fixed dataset, with no further interaction. It sounds like imitation learning but can improve over the behaviour in the data. The difficulty is distribution shift: the learned policy may choose actions the data never contained, and a value function can be wildly over-optimistic about them. The tutorial/review (2005.01643) surveys the problem.

A toy that shows the failure

Five actions; the true reward rises from action 0 to action 2 and then falls off a cliff (action 4 has reward −10). The logged data only ever tried actions 0, 1 and 2. A value function that is a straight line in the action fits the data perfectly and extrapolates the upward trend to the actions nobody tried:

def fit_q(
    actions: torch.Tensor,
    rewards: torch.Tensor,
    conservatism: float = 0.0,
    steps: int = 2000,
) -> torch.Tensor:
    """Fits Q(a) = w a + b by regression, optionally with a CQL-style penalty.

    The penalty ``conservatism * (logsumexp_a Q(a) - Q(a_data))`` pushes the values of
    all actions down and the values of logged actions back up, so actions absent from
    the data end up lower than data-supported ones. At ``conservatism`` 0 this is plain
    least squares.

    Returns:
        The values Q(0) ... Q(4).
    """
    w = torch.zeros(1, requires_grad=True)
    b = torch.zeros(1, requires_grad=True)
    optimizer = torch.optim.Adam([w, b], lr=0.05)
    for _ in range(steps):
        q_all = w * ACTIONS + b
        q_data = w * actions + b
        loss = ((q_data - rewards) ** 2).mean()
        loss = loss + conservatism * (torch.logsumexp(q_all, dim=0) - q_data.mean())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return (w * ACTIONS + b).detach()

The plain method is confidently wrong: its value for action 4 is 3.98, higher than for anything in the data, and the policy that maximises it picks the cliff. This is extrapolation error, and it gets worse, not better, when a policy is optimised against the value function, because optimisation seeks out exactly the actions where the value is most wrongly high.

Two families of remedies, both visible here:

• Be pessimistic about the unseen (CQL, below). The penalty $\log\sum_a e^{Q(a)} - Q(a_{\text{data}})$ lowers the values of all actions and raises the logged ones back. It avoids the cliff, but in this toy it overshoots: with a straight-line value function the penalty tilts the whole line downward, so the greedy choice becomes action 0 rather than the best logged action 2. Conservatism protects you; it is also a tuning knob that can cost performance.
• Stay inside the data (IQL-style, below). Only compare actions that appear in the dataset. Here that picks action 2 immediately. It cannot be fooled by extrapolation, because it never extrapolates, but it also cannot discover an improvement outside what was logged.

def in_sample_choice(actions: torch.Tensor, rewards: torch.Tensor) -> tuple[int, float]:
    """The behaviour-constrained alternative: only consider actions seen in the data.

    Average the logged reward of each action that appears and pick the best one. It
    cannot extrapolate beyond the data, which is exactly why it cannot be fooled by it.
    """
    seen = actions.unique()
    means = torch.stack([rewards[actions == a].mean() for a in seen])
    action = int(seen[means.argmax()])
    return action, true_reward(torch.tensor(float(action))).item()

• Conservative Q-Learning (2006.04779) learns a conservative value function, so the value of a policy under it lower-bounds its true value.
• Implicit Q-Learning (2110.06169) never evaluates actions outside the dataset, yet can improve substantially over the best behaviour in the data.
• Decision Transformer (2106.01345) recasts RL as sequence modelling with a transformer (the Transformers module).

For driving, logs are the abundant resource. ReCogDrive (2506.08052) fine-tunes a diffusion planner with RL in the NAVSIM non-reactive simulator, which replays logged scenes rather than interacting with a live world (see the Driving data module).

Check yourself

1. What does DPO replace in the RLHF pipeline?
2. In the DPO loss, what does the reference model's term do?
3. Why can a standard off-policy method fail on a fixed dataset?
4. In the toy, why does the plain value function pick the cliff, and why does a conservative penalty of 5 give up the best logged action?
5. Why can a chosen response's probability fall during DPO training?
6. What does DPO's implied reward $\beta \log(\pi/\pi_{\text{ref}})$ let you do without training a reward model?
7. What is the difference between offline RL and imitation learning on the same logs?

Quiz

Answer each question; feedback appears as soon as you choose.

Sources

• Ouyang et al., Training language models to follow instructions with human feedback, 2022.
• Rafailov et al., Direct Preference Optimization, 2023.
• Levine et al., Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, 2020.
• Kumar et al., Conservative Q-Learning, 2020.
• Kostrikov et al., Implicit Q-Learning, 2021.
• Chen et al., Decision Transformer, 2021.