Reasoning and chain of thought

What chain of thought is

A language model normally answers in one go. Chain of thought (CoT) has it write the intermediate steps first, then the answer. The original paper (2201.11903) shows that giving a few worked examples with intermediate reasoning steps improves large language models on arithmetic, commonsense and symbolic reasoning, and that this ability appears in sufficiently large models. A follow-up (2205.11916) found that simply adding "Let's think step by step" before the answer gives much of the benefit with no examples.

Why would writing steps help? The usual view: each generated token is one forward pass through a fixed-depth network (the Transformers module), so a model that must answer immediately has a fixed amount of computation per answer. Writing steps spends more tokens, and so more computation, and puts intermediate results into the context for later steps to use. It is a reading, not a proof.

Using more samples: self-consistency

If one reasoning path can go wrong, sample several and vote. Self-consistency (2203.11171) samples diverse reasoning paths instead of one greedy path and picks the answer that most paths agree on.

It works because correct paths tend to agree with each other while wrong paths tend to end in different wrong answers. The toy below makes that assumption explicit.

Exercise and reference code (PyTorch)

Each sampled path is right with some probability; otherwise it lands on one of four wrong answers chosen evenly. Tests are in libs/tutorial/tests/test_reasoning.py; run uv run --group tutorial python -m libs.tutorial.reasoning.

def majority_vote_accuracy(
    correct_prob: float,
    num_wrong: int,
    samples: int,
    trials: int = 4000,
    seed: int = 0,
) -> float:
    """Estimates how often the most common sampled answer is the correct one.

    Args:
        correct_prob: Chance that one sampled path reaches the correct answer.
        num_wrong: Number of distinct wrong answers; the remaining probability is
            spread evenly over them.
        samples: Number of reasoning paths sampled per question.
        trials: Number of simulated questions.
        seed: Random seed.

    Returns:
        The fraction of trials where the correct answer received the most votes.
        Ties are broken at random.

    Raises:
        ValueError: If ``correct_prob`` is outside [0, 1] or ``num_wrong`` < 1.
    """
    if not 0.0 <= correct_prob <= 1.0:
        raise ValueError(f"correct_prob must be in [0, 1], got {correct_prob}")
    if num_wrong < 1:
        raise ValueError(f"num_wrong must be at least 1, got {num_wrong}")
    generator = torch.Generator().manual_seed(seed)
    wrong = (1.0 - correct_prob) / num_wrong
    probs = torch.tensor([correct_prob] + [wrong] * num_wrong)
    answers = torch.multinomial(
        probs.expand(trials, -1), samples, True, generator=generator
    )
    votes = torch.nn.functional.one_hot(answers, num_wrong + 1).sum(dim=1).float()
    votes += 0.5 * torch.rand(votes.shape, generator=generator)  # random tie-break
    return (votes.argmax(dim=1) == 0).float().mean().item()

In our run, with one path right 40 percent of the time, the vote was right 0.39 of the time with 1 sample, 0.54 with 5 and 0.88 with 25.

Two caveats the toy hides. First, it assumes independent samples and wrong answers that spread out; real model errors are correlated (the same mistake repeated), so the gain is smaller. Second, if one wrong answer is more likely than the correct one, voting makes things worse: the test test_voting_does_not_help_when_one_wrong_answer_is_more_likely shows 0.3 against 0.7 falling toward zero. Voting amplifies whatever the majority is.

When voting fails: shared mistakes

The toy above lets wrong answers spread evenly. Real models repeat the same mistake: a common misconception, a misleading cue in the prompt. Let a fraction of the wrong paths all land on one specific wrong answer, and a single path is still right 40 percent of the time:

def correlated_vote_accuracy(
    correct_prob: float,
    shared_error: float,
    samples: int,
    trials: int = 4000,
    seed: int = 0,
) -> float:
    """Majority vote when wrong paths often make the same mistake.

    Each path is correct with probability ``correct_prob``. Otherwise, with probability
    ``shared_error`` it lands on one specific wrong answer (the common misconception),
    and with the rest it lands on one of four other wrong answers evenly.
    """
    generator = torch.Generator().manual_seed(seed)
    wrong = 1.0 - correct_prob
    probs = torch.tensor(
        [correct_prob, wrong * shared_error] + [wrong * (1 - shared_error) / 4] * 4
    )
    answers = torch.multinomial(
        probs.expand(trials, -1), samples, True, generator=generator
    )
    votes = torch.nn.functional.one_hot(answers, 6).sum(dim=1).float()
    votes += 0.5 * torch.rand(votes.shape, generator=generator)
    return (votes.argmax(dim=1) == 0).float().mean().item()


def best_of_n_accuracy(
    correct_prob: float,
    verifier_accuracy: float,
    samples: int,
    trials: int = 4000,
    seed: int = 0,
) -> float:
    """Best-of-n with an imperfect verifier that scores each sample.

    A verifier scores a correct sample above a wrong one with probability
    ``verifier_accuracy`` (0.5 is a coin flip, 1.0 is perfect). Each sample gets
    score = quality + noise, where the noise is set so that two samples are ordered
    correctly with that probability. The highest-scoring sample is returned.
    """
    generator = torch.Generator().manual_seed(seed)
    correct = torch.rand(trials, samples, generator=generator) < correct_prob
    # P(correct score > wrong score) = Phi(1 / (sigma sqrt(2))) = verifier_accuracy.
    normal = torch.distributions.Normal(0.0, 1.0)
    sigma = 1.0 / (2**0.5 * normal.icdf(torch.tensor(verifier_accuracy)).item())
    scores = correct.float() + sigma * torch.randn(trials, samples, generator=generator)
    chosen = scores.argmax(dim=1)
    return correct[torch.arange(trials), chosen].float().mean().item()

With half the wrong paths sharing a mistake, voting still helps but needs far more samples (0.89 at 101 against 1.00). With 80 percent, the shared wrong answer holds $0.6 \times 0.8 = 0.48$ of the probability against the correct answer's 0.40, so it becomes the plurality and more votes make things worse (0.20 at 101 samples). Voting converges on whatever answer is most likely; it does not know which is true. A test checks the reversal.

Spending test-time compute: verifiers

Voting needs no judge, only agreement. If you have a verifier that scores candidate answers (a reward model, a unit test, a rule check), you can sample $n$ candidates and return the best-scoring one (best-of-$n$). Everything then depends on the verifier:

A 90 percent verifier turns a 40 percent model into a 97 percent system with 25 samples, and a 55 percent verifier is nearly useless however many samples you draw. This is why checkable domains (mathematics, code, rule-based driving checks) benefit most from spending compute at test time, why verifier quality is itself a research topic ("Let's Verify Step by Step" trains process-level verifiers), and why reward-model over-optimisation (the RL for alignment module) is the risk: pushing $n$ up selects for samples the verifier likes, which are not always the ones that are right.

The budget view. Test-time compute is paid in latency and money. Voting over 25 samples is 25 times the decoding of one; for an onboard driving model with a 100 ms budget (Deploying a driving VLA) this is available only offline, for labelling and evaluation, or where sampling can run in parallel on spare hardware. Onboard, the equivalent is one carefully trained reasoning pass, which is why reasoning is trained with RL rather than sampled many times at inference.

Training a model to reason

• Supervised fine-tuning on reasoning traces. Train on examples that contain the reasoning and the answer. Cheap, but copies whatever the labels did, including mistakes (the Imitation learning module's imitation problem again).
• Outcome versus process supervision. Outcome supervision rewards only the final answer; process supervision scores each step. "Let's Verify Step by Step" (2305.20050) compares the two and reports that process supervision outperforms outcome supervision on the MATH problems it studies.
• Reinforcement learning on checkable answers. DeepSeek-R1 (2501.12948) reports that reasoning abilities can be incentivised by RL alone, with no human-labelled reasoning trajectories, on tasks whose answers can be verified (mathematics, coding competitions). The RL method is the group-relative approach from the Reinforcement learning module.

Reasoning and acting

ReAct (2210.03629) has a language model write reasoning traces and task actions interleaved: the reasoning helps track and update a plan and handle exceptions, and the actions fetch information from the environment. It is the template for agents that think, act, observe and think again.

Robot and driving models adopt the same idea in different forms:

Chain of Causation versus free-form chain of thought

Free-form CoT is whatever text the model writes before answering. Alpamayo-R1's Chain of Causation is structured: a driving decision chosen from a fixed list, tied to the causal factors in the scene (critical objects, traffic lights, road events). In the paper's evaluation (an LLM judge that agreed with human evaluation 92 percent of the time), structured traces improved the causal-relationship score by 132.8 percent over free-form traces, and a reward checks that the written reasoning agrees with the action actually taken (the Reinforcement learning module). The aim is reasoning the action depends on, not a story told afterwards.

Is the reasoning real? Faithfulness

A fluent explanation does not prove the model used it. Two studies on language models:

• Language Models Don't Always Say What They Think (2305.04388) finds that CoT explanations can systematically misrepresent the true reason for a prediction: biasing the input (for example reordering answer options so the answer is always "A") changed predictions while the explanations did not mention the bias.
• Measuring Faithfulness in Chain-of-Thought Reasoning (2307.13702) intervenes on the CoT (adding mistakes or paraphrasing it) and finds the model's dependence on its stated reasoning varies a lot between tasks, sometimes heavy and sometimes almost none.

For a driving model the lesson is practical: reading the reasoning trace is useful for debugging, but it is not evidence of why the trajectory came out as it did. The consistency reward and closed-loop testing exist because of this gap.

The cost: latency

Every reasoning token is another decoding step. The Alpamayo-R1 paper reports about 70 ms to decode 40 reasoning tokens (Table 14), roughly 1.75 ms per token, inside an end-to-end budget of about 100 ms. Longer reasoning means a slower car. That is why AutoVLA aims for adaptive reasoning, why the Alpamayo-R1 authors list on-demand reasoning as future work, and why the VLA structure in the VLA models module keeps the trajectory expert fast.

Check yourself

1. Why might writing out steps help a model that has a fixed number of layers?
2. Why does self-consistency help only when wrong answers disagree with each other?
3. What is the difference between outcome and process supervision?
4. Why is a plausible reasoning trace not proof that a driving model understood the scene?
5. At 1.75 ms per token, how many reasoning tokens fit in 35 ms?
6. A wrong answer holds 48 percent of the probability and the right one 40 percent. What happens to a majority vote as you add samples?
7. A verifier orders right and wrong samples correctly 55 percent of the time. What does best-of-100 achieve, and why?
8. Why is best-of-n usually an offline tool for an onboard driving model?

Quiz

Answer each question; feedback appears as soon as you choose.

Sources

Every paper above is linked inline; titles and identifiers were checked on arXiv. Alpamayo-R1 statements are from arXiv 2511.00088.