From modular stacks to end-to-end driving

Why this module

Most people who have shipped autonomous driving software built a modular stack: perception produces an object list and a map, prediction forecasts each agent, planning searches for a trajectory under rules and costs, and control tracks it. Each module has its own team, its own labels and its own tests. The driving models in the rest of this tutorial replace much of that with networks trained on driving data directly, and the job postings for them ask for different skills.

This module is the bridge. It maps each module you know onto its learned counterpart, measures on a toy problem why end-to-end learning wins and what it costs, and lists what carries over from a modular career.

1. A map from modules to learned components

Two things on the right-hand side are new rather than renamed. The interfaces between the boxes become learned features instead of hand-designed structs, and improvement comes mainly from data and training recipes instead of from code changes to one module.

2. Why end-to-end: the interface loses information

Every interface in a modular stack is a design decision about what downstream modules are allowed to know. An object list with position, size and velocity is a good summary, but it is a summary. Anything not in the struct (a turn signal, a pedestrian's head direction, a hand gesture from a traffic officer) is invisible to the planner unless someone adds a field, a label set and a model for it.

The toy in the reference code makes this concrete. A lead car sits ahead at some gap and lateral offset, and may show a cut-in cue. The expert brakes when the car is close and either in our lane or signalling into it. Three systems decide whether to brake:

• Modular: a perception network, trained on plenty of box labels, estimates the car's offset and gap; a hand-written rule brakes if the car is in our lane and close.
• End-to-end: one network from the raw sensor vector (16 channels mixing the scene, plus 16 channels of clutter) to the decision, trained on expert decisions.
• Hybrid: the modular perception stays, and a learned planner sees its object list and the raw sensor vector.

class Perception(nn.Module):
    """Raw sensor vector to an object list entry: (offset, gap / 10)."""

    def __init__(self, hidden: int = 64) -> None:
        """Creates the regressor."""
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(SENSOR, hidden),
            nn.ReLU(),
            nn.Linear(hidden, hidden),
            nn.ReLU(),
            nn.Linear(hidden, 2),
        )

    def forward(self, raw: torch.Tensor) -> torch.Tensor:
        """Returns the estimated (offset, gap / 10)."""
        return self.net(raw)


def train_perception(data: dict[str, torch.Tensor], steps: int = 1500) -> Perception:
    """Fits perception on box labels, which a modular programme has in bulk."""
    torch.manual_seed(0)
    model = Perception()
    optimizer = torch.optim.Adam(model.parameters(), lr=3e-3)
    target = torch.stack([data["offset"], data["gap"] / 10.0], dim=1)
    for _ in range(steps):
        loss = ((model(data["raw"]) - target) ** 2).mean()
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return model


@torch.no_grad()
def modular_stack(perception: Perception, raw: torch.Tensor) -> torch.Tensor:
    """Perception produces an object list; a hand-written rule plans from it.

    The object list has no field for the turn signal, so the planner cannot react
    to a car that is about to cut in. Each module can be tested on its own.
    """
    box = perception(raw)
    offset, gap = box[:, 0], box[:, 1] * 10.0
    return ((offset.abs() < 1.0) & (gap < 25.0)).long()

class EndToEnd(nn.Module):
    """A learned planner: input features to brake / no-brake logits."""

    def __init__(self, inputs: int, hidden: int = 64) -> None:
        """Creates the network for ``inputs`` input features."""
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(inputs, hidden),
            nn.ReLU(),
            nn.Linear(hidden, hidden),
            nn.ReLU(),
            nn.Linear(hidden, 2),
        )

    def forward(self, features: torch.Tensor) -> torch.Tensor:
        """Returns decision logits."""
        return self.net(features)


def planner_inputs(raw: torch.Tensor, perception: Perception | None) -> torch.Tensor:
    """The raw vector, plus perception's object list when ``perception`` is given."""
    if perception is None:
        return raw
    with torch.no_grad():
        return torch.cat([raw, perception(raw)], dim=1)


def train_planner(
    data: dict[str, torch.Tensor],
    perception: Perception | None,
    steps: int = 1500,
    seed: int = 0,
) -> EndToEnd:
    """Trains a learned planner on the expert labels."""
    torch.manual_seed(seed)
    features = planner_inputs(data["raw"], perception)
    model = EndToEnd(features.shape[1])
    optimizer = torch.optim.Adam(model.parameters(), lr=3e-3, weight_decay=1e-4)
    for _ in range(steps):
        loss = nn.functional.cross_entropy(model(features), data["brake"])
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    return model

(Accuracy, mean of 3 seeds; uv run --group tutorial python scripts/tutorial_results.py bridging.) Four lessons, each of which holds well beyond the toy:

• The modular stack is strong where its interface is enough. It needs no decision labels at all and is right 87 percent of the time.
• Its failures are structural. On cut-in scenes it is right 3 percent of the time, and no amount of driving data fixes that: someone has to change the interface.
• End-to-end pays for its flexibility in data. With 50 labelled scenes it is worse than the rules. It passes them somewhere between 100 and 200 scenes here; in real driving, where scenes are far more varied, the crossover can be expected to need far more data.
• Hybrids get the best of both when decision data is scarce. Feeding the learned planner the modular object list, which costs nothing extra, adds 1.4 to 2.5 points between 50 and 1000 scenes. With plenty of data the gap closes.

3. Hybrid stacks in the literature

End-to-end rarely means "no structure inside". Common patterns:

• Planning-oriented stacks with learned interfaces. UniAD (2212.10156) keeps tracking, mapping, motion and occupancy modules, connects them through query interfaces, and optimises the whole for planning. The modules remain, but the boundaries are learned features and gradients flow through them.
• Rule-based teachers. Hydra-MDP (2406.06978) distils from human and rule-based teachers, so the student learns how the environment shapes planning; the rule-based scorers you may have written become training signals (the Distillation module).
• Rule-based priors and baselines. "Parting with Misconceptions" (2306.07962) found that simple rule-based priors, such as a centerline from lane-graph search, add value in closed-loop planning; its simple planner won the nuPlan challenge in 2023.
• One multimodal model for many tasks. Waymo's EMMA (2410.23262) trains a large multimodal model to do tasks such as detection and motion planning jointly, with outputs expressed as text: the module outputs still exist, as tasks of one model.
• Reasoning on top. Alpamayo-R1 (2511.00088) adds a reasoning trace and a flow-matching trajectory decoder to a vision-language backbone (the Driving VLA concepts page).

4. What carries over from a modular career

More than people expect. Teams training driving foundation models hire for these, and engineers from modular stacks usually have them:

What is genuinely new, and what the rest of this group of modules covers:

• Improving a model through data and recipes, not code. When the planner misbehaves, the fix is usually more or better data for that situation, a loss term or a training change, and the test is an evaluation run, not a unit test.
• Training at scale: memory, parallelism, mixed precision and throughput (Training at scale).
• Scaling and data recipes: how loss falls with data, compute-optimal sizing, the long tail (Scaling laws and data recipes).
• Making large models small and fast: distillation (Distillation).
• Generative models and post-training: diffusion, flow matching, RL (Flow matching, RL from scratch).

5. Moving a stack over: practical patterns

These are common engineering practices, written as our recommendations rather than results from a paper:

• Shadow mode first. Run the learned planner alongside the modular one on the same inputs, log both outputs, and mine the disagreements; they are the most informative scenes you have.

• Keep the modular outputs as auxiliary tasks. Perception and prediction heads cost little, make the model debuggable, and, as the toy showed, help when decision data is scarce.
• Keep the rule-based planner as a fallback and a baseline. A safety filter or fallback is easier to argue for than a learned policy alone, and a baseline keeps evaluation honest.
• Change the unit of progress. Track metrics by scenario category (cut-ins, unprotected turns, construction) instead of by module, because the model has no modules to blame.

Reference implementation (PyTorch)

libs/tutorial/modular_e2e.py; tests are in libs/tutorial/tests/test_modular_e2e.py: the expert labels follow the rule, the object list hides the cut-in cue (the modular stack is below 10 percent on cut-ins but above 85 percent overall), end-to-end is worse than modular at 50 scenes and better by more than 3 points at 1000, and the hybrid beats end-to-end by more than 1.5 points at 200 scenes (about twice the seed-to-seed noise) and is at least as good at 1000. Run uv run --group tutorial python -m libs.tutorial.modular_e2e.

Check yourself

1. Name three interfaces in a modular stack and one piece of information each can lose.
2. Why can more driving data not fix the modular stack's cut-in failures in the toy?
3. Why does end-to-end need more labelled data than the modular planner, and what supplies that data in practice?
4. What does the hybrid gain from the object list, and why does the gain shrink with more data?
5. Which of your modular-stack skills map onto the data flywheel, evaluation and safety work of a foundation-model team?

Quiz

Answer each question; feedback appears as soon as you choose.

Sources

• Hu et al., Planning-oriented Autonomous Driving (UniAD), 2022.
• Li et al., Hydra-MDP, 2024.
• Dauner et al., Parting with Misconceptions about Learning-based Vehicle Motion Planning, 2023.
• Hwang et al., EMMA: End-to-End Multimodal Model for Autonomous Driving, 2024.
• NVIDIA et al., Alpamayo-R1, 2025.