Course: End-to-end driving
A course path through the tutorial for engineers who know modular driving stacks: from perception, prediction and planning modules to driving vision-language-action models like Alpamayo-R1.
Who this course is for
You have built or worked with a modular driving stack: perception produces an object list, prediction forecasts each agent, a planner searches under rules and costs, and a safety layer checks the result. The newest driving systems replace much of that with models trained end to end on driving data, and the newest of those are vision-language-action models (VLAs) that also reason in text. This course takes you from the first to the second. It starts from the stack you know, rebuilds each part as a learned component with code you can run, and ends with Alpamayo-R1, an open driving VLA, taken apart.
How each module works
Every module follows the same pattern:
- a box at the top with the study time, what you will be able to do afterwards and what to read first;
- the explanation, with figures and the PyTorch code it describes (each piece is tested, and the numbers quoted in the text come from running it);
- Check yourself questions, with worked answers you can open once you have tried them;
- a short quiz that gives feedback as soon as you choose.
The code runs on a laptop CPU with uv run --group tutorial python -m libs.tutorial.<module>.
Most modules also have a lab: the module's code with its key functions left for you to write, graded by the same tests that check the reference code. The syllabus lists each module's lab, and Course labs explains how they work. The course ends with a capstone: ship a learned car-following driver behind a safety envelope, graded by tests and a written safety case.
Background you need
You should be comfortable training a neural network with gradient descent and know what a loss, a batch and an embedding are. The course uses four machine-learning ideas throughout: attention, images and text as one token sequence, flow matching and policy gradients. Section 0 covers each in a short toolkit page with a quiz; each page links to the full module if you want the details.
Syllabus
About 23 hours of study in ten sections.
0. Toolkit
The four machine-learning ideas the rest of the course is built on, in short form. Skip any you know. About 2 hours.
| Module | Time | Lab |
|---|---|---|
| Toolkit: attention and transformers | 25 min | attention |
| Toolkit: images and text as one sequence | 25 min | |
| Toolkit: generating trajectories with flow matching | 25 min | flow_matching |
| Toolkit: rewards and policy gradients | 30 min | grpo |
1. From modular stacks to end-to-end
Why learned driving replaces hand-designed interfaces, and what carries over from a modular career. About 35 minutes.
| Module | Time | Lab |
|---|---|---|
| From modular stacks to end-to-end driving | 35 min | modular_e2e |
2. Perceiving the scene
How cameras, lidar, radar, maps and ego state become features, and how detections become tracks. About 2 hours.
| Module | Time | Lab |
|---|---|---|
| Sensor encoders for driving | 55 min | sensor_encoders |
| Camera geometry and object queries | 35 min | geometry |
| State estimation and tracking | 35 min | state_estimation |
3. Data and labels
Where driving data and labels come from, how to check them, and how to find the rare scenes. About 50 minutes.
| Module | Time | Lab |
|---|---|---|
| Driving data and labels | 50 min | label_quality |
4. Learning to drive from demonstrations
Imitation and its failure modes, vehicle models and planners, predicting other drivers, and giving the policy a route. About 3 hours.
| Module | Time | Lab |
|---|---|---|
| Imitation learning and action chunking | 50 min | imitation |
| Planning and vehicle models | 40 min | planning |
| Predicting other drivers | 40 min | |
| Route conditioning | 35 min |
5. Driving VLAs
Vision-language-action models, reasoning, Alpamayo-R1 taken apart in the paper and in the code, the line of papers that led to it, and how actions become tokens. About 3.5 hours.
| Module | Time | Lab |
|---|---|---|
| VLA models | 45 min | |
| Reasoning and chain of thought | 35 min | |
| Driving VLA concepts | 25 min | |
| Reading the Alpamayo-R1 code | 40 min | alpamayo_pieces |
| Driving VLA literature | 20 min | |
| Discrete tokenizers: VQ-VAE and FSQ | 45 min |
6. Beyond imitation
Reinforcement learning with driving rewards, world models that imagine the future, and keeping old skills while learning new ones. About 2.5 hours.
| Module | Time | Lab |
|---|---|---|
| Reinforcement learning for policies and reasoning | 65 min | policy_gradient |
| World models | 35 min | |
| Co-training and catastrophic forgetting | 40 min |
7. Testing a driving model
Metrics, open versus closed loop, simulation, and the data engine that turns failures into training data. About 80 minutes.
| Module | Time | Lab |
|---|---|---|
| Evaluating driving models | 35 min | driving_metrics |
| Closed-loop simulation and the data engine | 45 min |
8. Safety and shipping
Uncertainty and monitors, safety envelopes and evidence, and fitting the model into an onboard latency budget with adapters and quantization. About 3.5 hours.
| Module | Time | Lab |
|---|---|---|
| Uncertainty and runtime safety | 50 min | conformal |
| Safety engineering for learned drivers | 40 min | safety_engineering |
| Deploying a driving VLA: the onboard latency budget | 55 min | |
| Efficient fine-tuning and inference | 65 min |
9. Capstone
Build a mini driving VLA end to end, then ship a learned car-following driver with a safety envelope, graded by tests and a written safety case. About 4 hours.
| Module | Time | Lab |
|---|---|---|
| Project: build a mini driving VLA | 90 min | mini_driver |
| Capstone: ship a car-following driver | 150 min | capstone |
After the course
- Read the Alpamayo-R1 paper with the reading list questions beside you.
- Do the practices: predict-then-run experiments and, if you have a 24 GB GPU and access to the gated weights, the released model.
- Use the glossary as a reference.