Predicting other drivers

The question a planner asks

Before the ego car commits to a lane change or a merge, it needs to know what the cars around it will do over the next several seconds. Two facts make this hard. The answer is not a single path: a car approaching an intersection may go straight or turn, and the logs only show which one it did. And the answer is not independent across cars: when two cars approach the same point, one yielding is why the other goes.

The two sections that follow build a tiny version of each difficulty and measure it. The system view is in From modular stacks to end-to-end, where prediction is one module of the classic pipeline, and in the world-model module, where the same question is asked of whole scenes.

Difficulty 1: the future is multimodal

Take one car approaching a conflict point. It will either go (keep its speed) or yield (brake to a stop). Suppose the logs show it going 80 percent of the time in this situation. What should a predictor output?

• A predictor trained to minimise squared error converges to the average trajectory: 80 percent of the way between "goes" and "yields". That path is one the car never takes: in our example it crosses the conflict zone more slowly than a car that goes, and without stopping. It is the best single guess for the mean error and the worst one for the planner.
• A predictor that outputs several trajectories with probabilities (modes) keeps both futures. This is the idea behind MultiPath (1910.05449), which predicts probabilities over anchor trajectories plus offsets, and behind the attention-based predictors Wayformer (2207.05844) and the Motion Transformer (2209.13508), which output a set of modes with scores. The generative policies in the flow-matching and VLA modules solve the same problem by sampling.

The metric follows the design. minADE over $k$ takes the best of $k$ predicted trajectories against the one that actually happened, so a predictor is not punished for modes the car did not take. In our toy, over conflict scenes, the average trajectory had a mean error of 5.5 m; the best of the two modes had an error of 0.

Difficulty 2: agents are not independent

The scene is a crossing. Two cars approach the same point at 10 m/s. If their arrival times are within a few metres of each other, one has to give way, and in the demonstrations the closer one goes 80 percent of the time. Otherwise both just go. Four outcomes are possible for the pair, but the data contain only two of them in a conflict: A goes and B yields, or B goes and A yields.

def sample_scenes(n: int, seed: int = 0) -> tuple[torch.Tensor, torch.Tensor]:
    """Distances to the conflict point and the demonstrated (A goes, B goes) labels.

    Returns:
        Distances (n, 2) in metres, and labels (n, 2) of 0/1 for "goes".
    """
    generator = torch.Generator().manual_seed(seed)
    distance = 20.0 + 20.0 * torch.rand(n, 2, generator=generator)
    gap = distance[:, 0] - distance[:, 1]
    conflict = gap.abs() < CONFLICT_GAP
    closer_goes = torch.rand(n, generator=generator) < 0.8
    a_goes = torch.where(gap < 0, closer_goes, ~closer_goes)
    goes = torch.ones(n, 2, dtype=torch.bool)
    goes[conflict, 0] = a_goes[conflict]
    goes[conflict, 1] = ~a_goes[conflict]
    return distance, goes.long()


def collides(distance: torch.Tensor, goes: torch.Tensor) -> torch.Tensor:
    """True where both cars go and are in the conflict zone at overlapping times."""
    arrive = distance / SPEED
    overlapping = (arrive[:, 0] - arrive[:, 1]).abs() < 2 * ZONE / SPEED
    return (goes[:, 0] == 1) & (goes[:, 1] == 1) & overlapping


def trajectory(distance: torch.Tensor, goes: torch.Tensor) -> torch.Tensor:
    """Distance still to travel at each of HORIZON steps, shape (n, HORIZON).

    A car that goes keeps its speed. One that yields brakes at BRAKE m/s^2 to a stop.
    """
    t = torch.arange(1, HORIZON + 1) * DT
    go = distance[:, None] - SPEED * t
    stop_time = SPEED / BRAKE
    tb = t.clamp(max=stop_time)
    yield_ = distance[:, None] - (SPEED * tb - 0.5 * BRAKE * tb**2)
    return torch.where(goes[:, None] == 1, go, yield_)

Now compare two predictors. A marginal predictor gives each car its own probability of going, as a function of the scene. A joint predictor gives a probability to each of the four combinations. Train both on the same demonstrations.

def evaluate(model: Predictor, n: int = 4000, seed: int = 123) -> dict[str, float]:
    """Per-car accuracy and the scene-level behaviour of sampled futures.

    Returns:
        ``per_car_nll``: mean negative log-likelihood of each car's own demonstrated
        action; ``collision``: share of conflict scenes in which one sampled joint
        future has both cars colliding; ``deadlock``: share where both yield (a joint
        future that never occurs in the data).
    """
    distance, goes = sample_scenes(n, seed=seed)
    generator = torch.Generator().manual_seed(seed)
    p = model.go_probability(distance).clamp(1e-6, 1 - 1e-6)
    nll = -(goes * p.log() + (1 - goes) * (1 - p).log()).mean().item()
    conflict = (distance[:, 0] - distance[:, 1]).abs() < CONFLICT_GAP
    sampled = model.sample(distance, generator)
    return {
        "per_car_nll": nll,
        "collision": collides(distance, sampled)[conflict].float().mean().item(),
        "deadlock": (sampled.sum(-1) == 0)[conflict].float().mean().item(),
    }

The result, in our run:

For each car on its own the two are about equally good: ask either "does car A go?" and you get nearly the same answer. The difference is entirely in the combinations. A marginal predictor samples each car independently, so when each has a 50 percent chance of going it will produce "both go" a quarter of the time. In a conflict that future is a crash. "Both yield" is a deadlock that never occurs in the data. The joint predictor has seen only the legal pairs and so samples only them.

This is the main reason scene-level models exist. Scene Transformer (2106.08417) predicts the futures of all agents together and is built to support both marginal and joint predictions; the practical point is that metrics computed per agent, including minADE, cannot see this failure at all. Both predictors above have the same per-car quality. A planner that reasons about interaction (for example one that picks a yielding action because it expects the other car to go) needs the joint.

Putting it in a stack

• Conditional prediction. Ask "what will the others do if the ego car does this?" This is where prediction and planning meet: the ego plan is an input to the prediction, as in the planning module, and the question differs from the marginal "what will they do" in exactly the way the toy differs from its marginal predictor.
• How many modes? Typically a handful to a few dozen. Too few misses real futures; too many makes the planner weigh implausible ones. Mode probabilities should be calibrated, which the conformal-prediction ideas can check.
• How to read an average. If a predictor outputs one trajectory, ask what the training loss was. Squared error gives the mean, which can be a path nobody takes.
• Evaluation. Report a joint or scene-level metric next to per-agent minADE, and look at how often sampled futures collide with each other, as the table above does. The evaluation module covers the per-agent metrics.

Why this matters for a VLA

An end-to-end driving model never outputs explicit predictions, but it faces the same two difficulties inside its trajectory head. A regression head averages modes (see the action-head comparison in the VLA module: regression landed between the two squares for the ambiguous instruction), and a model that does not represent the other agents' responses to its own plan cannot tell a safe merge from one that requires the other driver to brake. When a paper reports a good open-loop minADE, ask both questions: does the head represent more than one future, and is the result evaluated with other agents that react (see closed-loop evaluation)?

Reference implementation (PyTorch)

Tests are in libs/tutorial/tests/test_multi_agent.py. They check that demonstrations contain no collisions or deadlocks, that both predictors are equally good per car, that only the marginal one samples collisions and deadlocks, and that one averaged trajectory is worse than the best of two modes. Run uv run --group tutorial python -m libs.tutorial.multi_agent.

def regression_versus_modes(
    model: Predictor, n: int = 2000, seed: int = 7
) -> dict[str, float]:
    """Error of one averaged trajectory versus the best of the two modes, car A only.

    The averaged trajectory is the probability-weighted mean of the "goes" and "yields"
    trajectories (what a squared-error regression converges to). Best-of-two keeps both
    and scores the closer one, as minADE does.
    """
    distance, goes = sample_scenes(n, seed=seed)
    conflict = (distance[:, 0] - distance[:, 1]).abs() < CONFLICT_GAP
    d, truth_goes = distance[conflict, 0], goes[conflict, 0]
    go = trajectory(d, torch.ones_like(truth_goes))
    stay = trajectory(d, torch.zeros_like(truth_goes))
    truth = torch.where(truth_goes[:, None] == 1, go, stay)
    p = model.go_probability(distance[conflict])[:, 0:1]
    mean = p * go + (1 - p) * stay
    best_of_two = torch.minimum(
        (go - truth).abs().mean(1), (stay - truth).abs().mean(1)
    )
    return {
        "averaged": average_displacement(mean, truth),
        "best_of_two": best_of_two.mean().item(),
    }

Check yourself

1. Why does a squared-error loss produce a trajectory that no driver takes?
2. What does minADE over $k$ reward, and what failure can it not see?
3. Two predictors have the same per-agent accuracy but one samples collisions. What is different about them?
4. A marginal predictor says each of two cars goes with probability 0.5 in a conflict. How often will it sample "both go"?
5. What is the difference between predicting what others will do and predicting what they will do given the ego plan?
6. Why is it risky to evaluate a planner against logged agents that never react?

Quiz

Answer each question; feedback appears as soon as you choose.

Sources

• Chai et al., MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction, 2019.
• Ngiam et al., Scene Transformer: A unified architecture for predicting multiple agent trajectories, 2021.
• Nayakanti et al., Wayformer: Motion Forecasting via Simple & Efficient Attention Networks, 2022.
• Shi et al., Motion Transformer with Global Intention Localization and Local Movement Refinement, 2022.
• The crossing numbers in this page are from our reference implementation, not from a dataset.