Driving VLA literature

The question driving research is answering

How should a car go from pixels to a trajectory, and where does language help? The papers below are best read as answers to five sub-questions.

1. Learn the whole stack, not parts

UniAD, "Planning-oriented Autonomous Driving" (2212.10156, Dec 2022; CVPR 2023 Best Paper). Instead of separate perception, prediction and planning models, UniAD connects transformer-decoder modules for tracking, mapping, motion and occupancy to a planner through unified query interfaces, optimising for the planning goal. Read it for the modular end-to-end baseline that language models later tried to replace or augment. Uses transformers (the Transformers module) but no language.

2. Treat planning as language modelling

GPT-Driver (2310.01415, Oct 2023) writes planner inputs and outputs as text and has an LLM produce trajectory coordinates as text. Read it for the simplest version of "actions as tokens" (the VLA models module) and its limit: numbers as text are an awkward trajectory format.

DriveGPT4 (2310.01412, Oct 2023) and LMDrive (2312.07488, Dec 2023) bring multimodal and language models into end-to-end driving, with interpretability (DriveGPT4) and closed-loop driving (LMDrive) as stated aims.

3. Use language for structured reasoning

DriveLM (2312.14150, Dec 2023; ECCV 2024 Oral) proposes Graph VQA: question-answer pairs about perception, prediction, planning, behaviour and motion, linked by dependencies, as a proxy for human multi-step reasoning, with a dataset on nuScenes and CARLA. Read it for the idea that reasoning should be structured and tied to the decision.

DriveVLM (2402.12289, Feb 2024; CoRL) has scene description, scene analysis and hierarchical planning stages. Its authors note VLMs are weak at spatial reasoning and expensive, so DriveVLM-Dual pairs it with a conventional pipeline, and they report deploying on a production vehicle. Read it for the practical tension between reasoning and real-time latency that every later paper addresses.

4. One large model for everything

EMMA (2410.23262, Waymo, Oct 2024) trains a large multimodal model on driving data to do tasks such as detection and motion planning jointly, with everything expressed as text. OpenDriveVLA (2503.23463, Mar 2025) builds a driving VLA on a Qwen-2.5 language model evaluated on nuScenes, with attention to aligning 2D and 3D features.

5. Generate continuous trajectories (diffusion and flow)

Text-coordinate planners (section 2) have format and feasibility problems. Generative planners fix this (the Diffusion and Flow matching modules).

DiffusionDrive (2411.15139, Nov 2024; CVPR 2025 Highlight) uses a truncated diffusion process starting from anchored noise so that a handful of denoising steps (2 in the authors' configuration) give diverse, high-quality trajectories in real time. Read it for how to make diffusion fast enough for a car.

ReCogDrive (2506.08052, Jun 2025) combines a VLM with a diffusion planner, injecting the VLM's driving knowledge into the planner. Its authors say language-modelling planners can produce format-violating or infeasible actions and are slow, and that a diffusion planner fixes this. Training includes reinforcement learning of the planner in the NAVSIM simulator.

6. Reasoning, causal labels and reinforcement learning

Alpamayo-R1 (2511.00088, NVIDIA, 2025) combines several of the lines above: a physical-reasoning VLM backbone (Cosmos-Reason1), Chain of Causation labels that tie decisions to causes, unicycle-style trajectory output, a flow-matching expert for continuous trajectories, and SFT followed by GRPO reinforcement learning with rewards on reasoning quality, reasoning-action consistency and safety. See Driving VLA concepts for details and the limits the authors state. AutoVLA (2506.13757, Jun 2025) puts reasoning and trajectory planning in one autoregressive model, with adaptive reasoning to avoid unnecessarily long reasoning and reinforcement fine-tuning; see Reasoning for the latency trade-off. GRPO is described in DeepSeekMath.

Surveys and maps

A Survey on Vision-Language-Action Models for Autonomous Driving (2506.24044, Jun 2025) with the Awesome-VLA4AD list. The field moves monthly; use it to locate newer work.

Patterns across the papers

Open questions that recur: whether reasoning text causes better driving or only accompanies it, how to evaluate safely beyond open loop, and how to get reliable labels at scale.

Comparing the papers on the same axes

Every row below restates what the sections above say; where a paper's abstract does not say, the cell is left blank.

How to read a driving paper

A checklist for the first pass, in the order that finds problems fastest (our recommendation, not from the papers):

1. What goes in, what comes out? Camera count and frames, whether route or ego state is given, what an output is (text, tokens, continuous waypoints, a distribution).
2. What was it trained on? Dataset, size, and whether the labels are human, machine-generated or free from logs (Driving data and labels).
3. How was it evaluated? Open loop on logs, non-reactive simulation, closed loop, or a real vehicle (Evaluating driving models). Which metrics, and is a rule-based baseline in the table?
4. What is the hard subset? An average hides rare cases; look for a separate long-tail result.
5. What does it cost? Parameters, tokens, latency on named hardware (Deploying a driving VLA).
6. What is not claimed? Safety cases, redundancy, generalisation to new cities: the limitations section is usually the most informative page.

A sensible reading path for the first week: UniAD (the modular baseline the rest react to), one language-planner paper (GPT-Driver or EMMA), DiffusionDrive (why continuous output), then Alpamayo-R1 (the combined recipe), spending about an hour each with the checklist above.

Check yourself

1. Which papers write trajectories as text, and what problem does DiffusionDrive or ReCogDrive address?
2. What does DriveVLM-Dual give up, and what does it keep?
3. How does Alpamayo-R1 differ from DriveLM in how reasoning is used?
4. Pick one paper, find its evaluation, and say whether it is open loop or closed loop.
5. Use the checklist on a paper not listed here: which of the six questions is hardest to answer from the abstract alone?

Answers

1. GPT-Driver writes trajectory coordinates as text, and EMMA expresses all its outputs as text. Text coordinates can violate the format, be infeasible and are slow to generate; DiffusionDrive and ReCogDrive generate continuous trajectories with a diffusion planner instead.
2. It gives up having the large VLM on the critical path for every decision; it keeps the VLM's scene description and analysis, and pairs it with a conventional pipeline for real-time planning.
3. DriveLM uses graph-structured question answering as a proxy for multi-step reasoning about the scene. Alpamayo-R1 generates a reasoning trace tied to the action it takes, and trains it with rewards for reasoning quality and for consistency between reasoning and action.
4. Example: DiffusionDrive and ReCogDrive report results in NAVSIM, which is non-reactive simulation over a short horizon; Alpamayo-R1 reports closed-loop simulation in AlpaSim plus on-vehicle road tests. Check the evaluation section, not the abstract.
5. Usually question 4 (the hard subset) or 6 (what is not claimed): abstracts report averages and rarely say what was left out.

Quiz

Answer each question; feedback appears as soon as you choose.

Sources

Every paper above is linked inline; titles and identifiers were checked on arXiv.