Closed-loop simulation and the data engine

Two questions this module answers

1. How do you test a driving policy before it drives? Open-loop metrics compare its output with a log, which says little about what happens once its own actions change the world (see Evaluating driving models). You need to run the policy in a world that responds. That is closed-loop simulation.
2. How do you make the policy better after it fails? Collecting more data at random mostly returns more of what the model already handles. A data engine finds the failures and turns them into training data.

The two fit together: the simulator is how the engine finds failures cheaply, and the engine is what makes the simulator's findings change the model.

What a simulator has to get right

Every closed-loop simulator replaces three things the real world provides.

Public examples: CARLA (1711.03938) for open-source urban driving; Bench2Drive (2406.03877), a multi-ability closed-loop benchmark built on it; nuPlan (2106.11810), a closed-loop planning benchmark on real logs, where agents can be replayed or controlled by a reactive model; and AlpaSim, the simulator the Alpamayo-R1 paper evaluates in (see the data module). A policy that takes camera images needs sensors realistic enough that its perception transfers, which is why simulators for end-to-end models invest in rendering or reconstruction (the sim-to-real module covers the gap that remains).

Why replaying logs gives wrong answers

Log replay is the cheapest simulator: put the ego car in a recorded scene and play the other agents back exactly as recorded. It is also the one that most easily produces misleading numbers, because the other agents cannot react to what the ego car does differently from the log.

A concrete case. In the log, the original ego car brakes gently at 2 m/s² for 3 s, with a following car 15 m behind at the same speed. Test a new policy that brakes at 4 m/s² in the same situation. A replayed follower repeats its logged speeds, so it keeps moving as if the old, gentler ego were still in front. It runs into the new ego car. A reactive follower, one that uses a time-gap controller, slows down and nothing happens.

def follower_gaps(steps: int = 60) -> tuple[torch.Tensor, torch.Tensor]:
    """Gap from a following car to a new ego car that brakes harder than the logged one.

    The log: the original ego car and a follower 15 m behind, both at 10 m/s, where the
    original ego brakes gently (2 m/s^2) for 3 s. The new ego brakes at 4 m/s^2 for 3 s.
    A replayed follower repeats its logged speeds, which were the original ego's; a
    reactive follower reuses the time-gap controller from the expert.

    Returns:
        The replayed and the reactive follower's gap to the new ego car at each step.
    """
    ego_speed, ego_position = torch.tensor(10.0), torch.tensor(0.0)
    logged_speed = torch.tensor(10.0)
    replay_position = torch.tensor(-15.0)
    reactive_speed, reactive_position = torch.tensor(10.0), torch.tensor(-15.0)
    replay_gaps, reactive_gaps = [], []
    for step in range(steps):
        braking = step < 30
        logged_speed = (logged_speed - DT * (2.0 if braking else 0.0)).clamp(min=0.0)
        ego_speed = (ego_speed - DT * (4.0 if braking else 0.0)).clamp(min=0.0)
        ego_position = ego_position + DT * ego_speed
        replay_position = replay_position + DT * logged_speed
        gap = ego_position - reactive_position
        accel = expert_action(gap, reactive_speed, ego_speed)
        reactive_speed = (reactive_speed + DT * accel).clamp(min=0.0)
        reactive_position = reactive_position + DT * reactive_speed
        replay_gaps.append(ego_position - replay_position)
        reactive_gaps.append(ego_position - reactive_position)
    return torch.stack(replay_gaps), torch.stack(reactive_gaps)


def replay_versus_reactive() -> dict[str, bool]:
    """Whether the replayed and the reactive follower hit the new ego car."""
    replay, reactive = follower_gaps()
    return {
        "replay_collision": bool((replay <= 0).any()),
        "reactive_collision": bool((reactive <= 0).any()),
    }

In our run the replayed follower collides (its gap crosses zero after 4.5 s) and the reactive one does not. The collision is real in the simulator and meaningless on the road: it tests whether the ego car drives like the logged one, not whether it drives well. Reactive agents fix this, at the price of making the simulator a model of other drivers, which can be wrong in its own ways (too polite, too aggressive, too uniform). Two practical consequences:

• Report which agents were reactive next to every closed-loop number. A collision rate with replayed agents and one with reactive agents are different measurements.
• Classify collisions by fault. Many benchmarks separate at-fault collisions from ones where the ego car was hit from behind, for exactly this reason.

A car-following toy for the data engine

To study the engine we need a policy that fails in an understandable way. The world is a lead car and an ego car on a line. The lead usually cruises with small speed changes; 2 percent of scenarios have a lead that brakes hard (5 m/s² for up to 3 s).

def make_scenarios(n: int, hard_share: float, seed: int) -> dict[str, torch.Tensor]:
    """Initial gap, speeds and a lead car that sometimes brakes hard.

    A hard-braking lead brakes at 5 m/s^2 for up to 3 s starting between 2 and 6 s in.
    Otherwise it keeps its speed with small random changes.
    """
    generator = torch.Generator().manual_seed(seed)
    speed = 8.0 + 6.0 * torch.rand(n, generator=generator)
    return {
        "gap": 15.0 + 25.0 * torch.rand(n, generator=generator),
        "ego_speed": speed,
        "lead_speed": speed + torch.randn(n, generator=generator),
        "hard": torch.rand(n, generator=generator) < hard_share,
        "brake_start": 2.0 + 4.0 * torch.rand(n, generator=generator),
        "wiggle": 0.3 * torch.randn(n, STEPS, generator=generator),
    }


def lead_accel(scenes: dict[str, torch.Tensor], step: int) -> torch.Tensor:
    """The lead car's acceleration at a time step."""
    t = step * DT
    braking = (
        scenes["hard"]
        & (t >= scenes["brake_start"])
        & (t < scenes["brake_start"] + 3.0)
    )
    return torch.where(
        braking, torch.full_like(scenes["gap"], -5.0), scenes["wiggle"][:, step]
    )

The expert is a time-gap controller that also brakes at full strength when the time to collision (gap divided by closing speed) is under 2 s. It never collides on any of the scenarios here.

def expert_action(
    gap: torch.Tensor, speed: torch.Tensor, lead: torch.Tensor
) -> torch.Tensor:
    """A time-gap controller with an emergency brake.

    It aims for a gap of 3 m plus 0.7 s of travel, and brakes at full strength when the
    time to collision (gap over closing speed) falls under 2 s.
    """
    wanted = 3.0 + 0.7 * speed
    accel = 0.4 * (gap - wanted) + 1.0 * (lead - speed)
    closing = (speed - lead).clamp(min=1e-3)
    emergency = (gap / closing < 2.0) & (speed > lead)
    accel = torch.where(emergency, torch.full_like(accel, -MAX_BRAKE), accel)
    return accel.clamp(-MAX_BRAKE, MAX_ACCEL)

A small network is trained by behaviour cloning on the expert's driving in 100 ordinary scenarios, as if the fleet had not yet met a hard-braking lead. Then it is tested in closed loop on 600 fresh scenarios, 30 percent of which have a hard-braking lead. It follows ordinary traffic without a single collision, and collides in about a quarter of all the test scenarios (28 percent in the first seed, 14 to 28 across four). Every failure is a hard-braking lead: the network had no reason to learn the emergency brake. This is the long tail from the data module in its smallest form.

The loop

The data engine is a loop with five steps.

1. Run the current policy in closed loop on a large pool of fresh scenarios.
2. Find the failures: collisions, hard-brake events, large deviations from a reference.
3. Label: the expert produces the correct action for the states the policy actually visited. This is the same idea as DAgger (1011.0686) from the imitation module: the policy picks the states, the expert picks the actions.
4. Add the new examples to the training set.
5. Retrain and re-measure, on a held-out test set that is never mined.

def data_engine(
    rounds: int = 3,
    targeted: bool = True,
    pool_size: int = 4000,
    added: int = 3,
    seed: int = 0,
) -> list[float]:
    """Runs the loop and returns the failure rate after each round (index 0 = start).

    Start by cloning expert driving on 100 ordinary scenarios (no hard-braking leads, as
    if the fleet had not met one yet); fresh scenarios have 2 percent.
    Each round the learner drives a pool of fresh scenarios; ``added`` of them are
    chosen (``targeted``: from the ones that ended in a collision; otherwise at random),
    the expert labels the states the learner visited, and the policy is refit on
    everything.
    """
    torch.manual_seed(seed)
    base = make_scenarios(100, 0.0, seed=seed)
    _, (f, y) = rollout(base, None)
    features_, labels = f.reshape(-1, 3), y.reshape(-1)
    policy = make_policy()
    fit(policy, features_, labels)
    history = [failure_rate(policy)]
    for round_ in range(rounds):
        pool = make_scenarios(pool_size, 0.02, seed=seed * 100 + round_ + 1)
        crashed, _ = rollout(pool, policy)
        if targeted:
            pick = crashed.nonzero().squeeze(-1)[:added]
        else:
            pick = torch.randperm(pool_size)[:added]
        _, (f, y) = rollout(subset(pool, _mask(pool_size, pick)), policy)
        features_ = torch.cat([features_, f.reshape(-1, 3)])
        labels = torch.cat([labels, y.reshape(-1)])
        fit(policy, features_, labels)
        history.append(failure_rate(policy))
    return history


def _mask(n: int, index: torch.Tensor) -> torch.Tensor:
    mask = torch.zeros(n, dtype=torch.bool)
    mask[index] = True
    return mask

The experiment compares two ways of choosing which scenarios to add. In both, three scenarios are added per round, from a pool of 4,000 with 2 percent hard-braking leads, and the expert labels every state the policy visited.

• Targeted: the three scenarios are taken from those where the policy collided.
• Random: the three scenarios are chosen at random from the pool.

After one round, the targeted loop has no collisions in all four seeds; the random loop is at about the starting rate. Ordinary traffic still has zero collisions afterwards, so nothing was traded away. Three random scenarios contain a hard-braking lead about once in 17, so random data mostly repeats ordinary driving; the failing scenarios, by construction, all contain the case. That is the whole argument for the engine: the value of a training example depends on what the model currently gets wrong.

Do not read the size of the effect as typical. Here the failure region is a single, simple event and three examples cover it. Real failure modes are many, overlapping and hard to describe, and a real engine runs many rounds with thousands of examples. The qualitative lesson (target, do not sample) holds; the speed does not.

What a real engine needs

• A trustworthy simulator. If the simulator's agents are unrealistic, the engine learns to beat the simulator. Spot-check mined failures against real logs.
• Failure mining that is not only collisions. Hard braking, near misses, traffic-rule violations, disengagements, and disagreement between model and expert all find scenarios worth labelling. Rarity scores like those in the driving data module find candidates before the policy has failed on them.
• Deduplication. One failure mode appears in thousands of near-identical scenarios. Adding all of them over-weights it; cluster first and sample from each cluster.
• A held-out set that stays held out. Mining the test set and then training on what it finds makes the test number meaningless. Keep a fixed evaluation set and, ideally, a fresh one for each release.
• Regression tests. A fix that breaks something else is common. Keep a set of previously fixed scenarios and re-run it on every candidate model.
• Labelling cost and quality. Where the expert is a human, each round costs labelling time, and the label-quality checks in the driving data module apply.
• Data mixing. New data competes with old. The scaling and data module covers up-weighting versus mining and why you should not simply repeat a rare example many times, and co-training covers how the new data can make the old capabilities worse.

Why this matters for a VLA

A VLA is trained mostly by imitation on logs, so it inherits the open-loop and long-tail problems this module describes. The Alpamayo-R1 paper uses a closed-loop simulator (AlpaSim) for its close-encounter results and picks RL training samples where the model's output disagrees most with a reward-based target (see the driving data module), which is the same idea as the engine: spend training effort where the model is weakest. When reading any driving-model paper, ask whether the headline number was measured in a world that reacts, and what the other agents were.

Reference implementation (PyTorch)

Tests are in libs/tutorial/tests/test_closed_loop.py. They check that the expert never collides even with a hard-braking lead, that it brakes at full strength when a collision is imminent, that the replayed follower collides with a new ego car and the reactive one does not, and that three failing scenarios remove the policy's failures where three random ones do not. Run uv run --group tutorial python -m libs.tutorial.closed_loop.

Check yourself

1. Why can a replayed other agent produce a collision that is not the ego car's fault?
2. What is the difference between a replayed, a rule-based reactive and a learned other agent, and what can each get wrong?
3. In the data engine, who chooses the states and who chooses the actions?
4. Three random scenarios did not help, and three failing ones did. What does that say about the value of a training example?
5. Why must the test set stay out of the mining loop?
6. Why is the toy's result (one round is enough) not a prediction for a real driving system?

Quiz

Answer each question; feedback appears as soon as you choose.

Sources

• Dosovitskiy et al., CARLA: An Open Urban Driving Simulator, 2017.
• Jia et al., Bench2Drive, 2024.
• Caesar et al., NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles, 2021.
• Ross, Gordon and Bagnell, A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, 2010 (DAgger).
• The toy results (28 percent starting collision rate in seed 0, 14 to 28 across four seeds, zero after one targeted round, about a quarter after random rounds) are from our reference implementation.