Evaluating driving models

Why this matters

A driving paper is a table of numbers. Whether a result means anything depends on what the metric rewards. This module covers the metrics a driving VLA is reported with, what each one cannot see, and the three evaluation settings (open loop, non-reactive simulation, closed loop), so you can read the Alpamayo-R1 tables, and your own, critically.

1. Displacement errors

The first metrics compare a predicted path with the logged one. For predicted points $\hat p_t$ and logged points $p_t$ over $T$ steps:

ADE is the average distance; FDE is the distance at the end. For models that output $k$ candidate paths, minADE$_k$ and minFDE$_k$ take the best candidate. AR1 reports minADE$_6$.

def ade_fde(prediction: torch.Tensor, truth: torch.Tensor) -> tuple[float, float]:
    """Average and final displacement error of one predicted path.

    Args:
        prediction: Shape (T, 2).
        truth: Shape (T, 2).

    Returns:
        Mean distance over all points, and distance at the last point, in metres.

    Raises:
        ValueError: If the shapes differ.
    """
    if prediction.shape != truth.shape:
        raise ValueError(
            f"shapes differ: {tuple(prediction.shape)} vs {tuple(truth.shape)}"
        )
    distances = (prediction - truth).norm(dim=-1)
    return distances.mean().item(), distances[-1].item()


def min_ade_fde(predictions: torch.Tensor, truth: torch.Tensor) -> tuple[float, float]:
    """minADE and minFDE over k predicted paths: the best of k on each metric.

    Args:
        predictions: Shape (k, T, 2).
        truth: Shape (T, 2).
    """
    distances = (predictions - truth).norm(dim=-1)
    return distances.mean(dim=1).min().item(), distances[:, -1].min().item()

Displacement error assumes the logged path is the right answer. In driving there are many acceptable paths, and the log is only one of them.

The average of two good answers is a bad answer

Suppose the logged driver passed a parked car on the left in half of the training scenes and on the right in the other half. An L2-trained regressor learns the average: straight through the car.

In the reference code the average has a mean distance of 1.23 m to the left log, exactly what a model that picks a side at random gets in expectation (0 m when right, 2.46 m when wrong), and a test checks the two are equal. A collision check separates them: the average collides, each logged path does not. minADE$_k$ over both modes would give 0, because it rewards covering the modes, but it then says nothing about which candidate the system would actually drive.

2. Collisions, comfort, and a combined score

Safety metrics check the predicted path against the world. A collision is the ego path coming within a radius of another road user at the same time. Comfort is measured from the smoothness of the path, for example the largest jerk (the third time derivative of position). The close-encounter rate in the glossary is the same idea as a collision check with a safety margin; the Alpamayo-R1 paper reports it in closed-loop simulation.

def has_collision(
    path: torch.Tensor, obstacles: torch.Tensor, radius: float = 1.5
) -> bool:
    """True if the path comes within ``radius`` of any obstacle point at the same time.

    Args:
        path: Ego positions, shape (T, 2).
        obstacles: Obstacle positions over time, shape (T, m, 2). A static obstacle is
            the same row repeated.
        radius: Collision distance in metres.
    """
    distances = (obstacles - path[:, None]).norm(dim=-1)
    return bool((distances < radius).any())


def max_jerk(path: torch.Tensor, dt: float = DT) -> float:
    """Largest jerk magnitude (m/s^3) from the third difference of positions.

    Raises:
        ValueError: If there are fewer than four points.
    """
    if len(path) < 4:
        raise ValueError(f"need at least 4 points, got {len(path)}")
    jerk = path.diff(n=3, dim=0) / dt**3
    return jerk.norm(dim=-1).max().item()


def pdm_score(
    no_collision: float,
    drivable: float,
    progress: float,
    time_to_collision: float,
    comfort: float,
    weights: tuple[float, float, float] = (5.0, 5.0, 2.0),
) -> float:
    """Aggregates sub-scores as the NAVSIM benchmark does: penalties times an average.

    The two penalty scores multiply the whole result, so a collision or leaving the
    road zeroes the score however good the rest is. Progress, time to collision and
    comfort are combined as a weighted average.

    Args:
        no_collision: Score in [0, 1]; 0 for an at-fault collision.
        drivable: Score in [0, 1]; 0 for leaving the drivable area.
        progress: Score in [0, 1].
        time_to_collision: Score in [0, 1].
        comfort: Score in [0, 1].
        weights: Weights for progress, time to collision and comfort. These are
            illustrative; the benchmark's configuration sets the real ones.

    Raises:
        ValueError: If a score is outside [0, 1] or the weights do not sum to a
            positive number.
    """
    scores = (no_collision, drivable, progress, time_to_collision, comfort)
    if any(not 0.0 <= score <= 1.0 for score in scores):
        raise ValueError(f"scores must be in [0, 1], got {scores}")
    if sum(weights) <= 0:
        raise ValueError(f"weights must sum to a positive number, got {weights}")
    average = (
        weights[0] * progress + weights[1] * time_to_collision + weights[2] * comfort
    ) / sum(weights)
    return no_collision * drivable * average

The NAVSIM benchmark (2406.15349) combines sub-scores into one number, the PDM score. Sub-scores in $[0,1]$ are aggregated as

with penalties for at-fault collisions (NC) and leaving the drivable area (DAC), and a weighted average of ego progress (EP), time to collision (TTC) and comfort (C). A penalty of 0 sets the whole score to 0. The weights in the reference code (5, 5, 2) are illustrative: the paper says the aggregation parameters can be changed, so use the benchmark's configuration for real numbers. The structure is the point: safety is a gate, not one more term to trade off. A model cannot buy back a collision with good comfort.

3. Open loop, non-reactive simulation, closed loop

Why open loop can mislead: the model is evaluated at states a human driver visited. In use it creates its own states. A tiny systematic error then compounds.

def open_vs_closed_loop(
    bias: float, steps: int = 40, speed: float = 10.0
) -> tuple[float, float]:
    """Error of a model with a constant heading bias per step, measured two ways.

    The true driver goes straight. The model's predicted heading change is wrong by
    ``bias`` radians at every step. Open loop: at each step the model starts from the
    logged state, so its error is the one step's displacement error. Closed loop: the
    model drives from its own state, heading errors accumulate, and the error is the
    final lateral offset.

    Returns:
        Mean open-loop one-step error and closed-loop final lateral offset, in metres.
    """
    open_loop = speed * DT * math.sin(bias)
    heading, lateral = 0.0, 0.0
    for _ in range(steps):
        heading += bias
        lateral += speed * DT * math.sin(heading)
    return abs(open_loop), abs(lateral)

In the reference code a per-step bias of 0.002 rad costs 0.002 m of open-loop error at 10 m/s, and the closed-loop offset grows with the square of time, because each step's heading error carries into every later step. The bias is chosen to be invisible in an open-loop table.

The NAVSIM paper makes the same point from the other side: open-loop evaluation with real data is easy, but those results do not reflect closed-loop performance, and a non-reactive simulation scored on progress and time to collision aligns better with closed-loop results than displacement errors do. "Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?" (2312.03031) argues that on nuScenes, models use ego status such as speed to plan and underuse perception, and that current metrics do not fully measure planning quality; it proposes an on-road check and a simple baseline. Read it before trusting an open-loop table.

4. Evaluating language: reasoning and question answering

A VLA that explains its decisions needs a metric for text, and word-overlap scores (BLEU, CIDEr) say little about whether an explanation is true. LingoQA (2312.14115) is a driving question-answering benchmark with 28K short video scenarios and 419K annotations. Its authors propose Lingo-Judge, a truthfulness classifier that reaches a 0.95 Spearman correlation with human judgments and beats METEOR, BLEU, CIDEr and GPT-4 as evaluators. Alpamayo-R1 uses a learned judge for reasoning quality too (the Reasoning module); the caveat is the same as for any learned judge: it is itself a model, and a fluent trace is not proof of a faithful one.

5. A checklist for reading a results table

• Which setting is it: open loop, non-reactive simulation or closed loop?
• Is a rule-based baseline in the table (for example the centerline planner of the previous module)?
• Are there safety numbers (collision, off-road) next to the displacement numbers?
• Is the hard subset reported separately from the average? AR1 reports both a nominal and a challenging long-tail set; an average can hide rare-case behaviour.
• Does the model have ego status as an input, and does an ego-status-only baseline score well?
• How many seeds or scenes, and is there an error bar?

Reference implementation (PyTorch)

libs/tutorial/driving_metrics.py; tests are in libs/tutorial/tests/test_driving_metrics.py: ADE and FDE on a constant offset, FDE looks only at the last point, minADE takes the best candidate, a collision needs closeness at the same time, constant velocity has zero jerk and a kink does not, a penalty zeroes the score, the weights set how much each sub-score matters, the average of two manoeuvres collides while each logged path does not, ADE cannot separate the average from picking a side, and a small bias is tiny open loop but large closed loop and grows faster than linearly. Run uv run --group tutorial python -m libs.tutorial.driving_metrics.

Check yourself

1. Why does minADE$_6$ reward hedging, and what does it not tell you?
2. Why do the PDM score's collision and off-road terms multiply rather than add?
3. How can a model look perfect open loop and still drift off the road?
4. What are the two things a learned judge adds and costs when evaluating reasoning text?

Quiz

Answer each question; feedback appears as soon as you choose.

Sources

• Dauner et al., NAVSIM, 2024.
• Li et al., Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?, 2023.
• Dauner et al., Parting with Misconceptions about Learning-based Vehicle Motion Planning, 2023.
• Marcu et al., LingoQA, 2023.