Toolkit: images and text as one sequence

Why this is in a driving course

A driving VLA starts life as a vision-language model (VLM): a model that reads images and text and writes text. Everything that makes it drive (ego history, trajectory tokens, an action expert) is added around that core. This page explains the core. The full treatment is the Tokens, ViT and VLMs module.

Text becomes tokens

A language model does not read characters. A tokenizer splits text into pieces from a fixed vocabulary (frequent words whole, rare words in parts), and each piece is an integer id. An embedding table turns each id into a vector. From then on, a sentence is a sequence of vectors, which is what a transformer (attention toolkit) reads.

The vocabulary is just a table, so it can be extended: add new entries, give each a new embedding, and train. Keep this in mind for the driving section.

Images become tokens

A Vision Transformer (ViT) cuts an image into square patches (16 by 16 pixels, say), flattens each patch and projects it to a vector: one token per patch, plus a position so the model knows where each patch came from. A 224 by 224 image with 16-pixel patches is 14 × 14 = 196 tokens.

One sequence

A VLM has three parts: a vision encoder that turns the image into patch tokens, a projector (often one linear layer) that maps those tokens into the language model's vector space, and the language model, which reads image tokens and text tokens in a single sequence. Every token can attend to every other, so a word in the question can attend to the patch that answers it.

The tutorial's tiny VLM is the whole idea in a few lines:

class TinyVLM(nn.Module):
    """Vision encoder, projector, and a transformer over image and text tokens."""

    def __init__(self, dim: int = 64, vision_dim: int = 48) -> None:
        """Creates the parts.

        Args:
            dim: Feature size of the language-model side.
            vision_dim: Feature size of the vision encoder's tokens.
        """
        super().__init__()
        self.vision = TinyViT(image_size=IMAGE, patch_size=4, dim=vision_dim, depth=1)
        self.projector = nn.Linear(vision_dim, dim)  # vision space -> language space
        self.question = nn.Embedding(len(QUESTIONS), dim)
        self.answer_slot = nn.Parameter(torch.zeros(1, 1, dim))
        self.blocks = nn.ModuleList(TransformerBlock(dim, heads=4) for _ in range(2))
        self.norm = nn.LayerNorm(dim)
        self.head = nn.Linear(dim, len(COLOURS))

    def forward(self, images: torch.Tensor, questions: torch.Tensor) -> torch.Tensor:
        """Returns answer logits (batch, 2) read from a learned answer token."""
        image_tokens = self.projector(self.vision(images))
        text_tokens = self.question(questions)[:, None]
        slot = self.answer_slot.expand(len(images), -1, -1)
        tokens = torch.cat([image_tokens, text_tokens, slot], dim=1)
        for block in self.blocks:
            tokens = block(tokens)
        return self.head(self.norm(tokens[:, -1]))

Where does the language knowledge come from? Real VLMs start from a language model pretrained on web text and a vision encoder pretrained to match images with their captions (contrastive training such as CLIP), then learn the projector and fine-tune on image-text pairs. A driving team inherits all of that, which is the argument for starting from a VLM at all.

The token budget

Image tokens are the expensive part. Many cameras times several frames times hundreds of patches is thousands of tokens before any text, and attention's cost grows with the square of that.

The released Alpamayo-R1 code reads four cameras with four frames each (16 images) and bounds the pixels per image, which bounds the tokens. Compressing or pruning image tokens is one of the main levers in the deployment module.

From a VLM to a driving model

Three additions turn a VLM into a driving VLA, and each is a variation on "more kinds of tokens":

1. Ego motion as tokens. The car's recent path is quantised into tokens and placed in the prompt. In the released Alpamayo-R1 code, 48 placeholder tokens hold 16 past positions in three coordinates (the code walkthrough).
2. Actions as tokens, or as an expert. Either the vocabulary is extended with trajectory tokens the model learns to write, or a separate action expert generates continuous actions while attending to the VLM's features (VLA models). Alpamayo-R1 trains with the first and drives with the second.
3. Special tokens for structure. Markers such as the start of the reasoning or of the future trajectory tell the model which part of the sequence it is in, and tell the code where to stop.

Where this shows up in the course

• Sensor encoders: cameras to tokens, and the cost per camera.
• VLA models: discrete action tokens against an action expert.
• Reasoning: the language model writing a trace before the action.
• The Alpamayo-R1 code: the prompt, the history tokens and the special tokens.

Check yourself

1. How many tokens does a 448 by 448 image make with 14-pixel patches?
2. What does the projector do, and why is it needed?
3. Why can a word in the question attend to an image patch?
4. What does it take to add a new kind of token, such as a trajectory token, to a language model?
5. Why does a driving VLA start from a pretrained VLM instead of training from scratch?

Quiz

Answer each question; feedback appears as soon as you choose.

Go deeper

The Tokens, ViT and VLMs module covers byte-pair encoding, ViT, CLIP and SigLIP, and the tiny VLM in detail, with measured results.