"A model understands action when the world changes in the intended way."
A Careful Robot Foundation Model
Vision-Language-Action models connect the semantic breadth of vision-language models with the physical discipline of robot policies. This chapter teaches how VLAs represent observations, instructions, robot state, and actions, then shows how the field moved from RT-1 and RT-2 to OpenVLA, Octo, RDT-1B, pi-zero, FAST, SmolVLA, GR00T, and Gemini Robotics.
A VLA is not a smarter captioner. It is an embodied policy whose output must be safe, timely, measurable, and recoverable inside the perception-action loop.
Chapter Overview
Chapter 34 is the bridge between the language and vision chapters and the robot foundation model chapter that follows. It starts from the core interface question, then studies the historical lineage, open policies, action heads, tokenization, co-training, prompting, and evaluation.
The chapter uses the right-tool pattern throughout. We build small contracts and numeric examples by hand, then show where LeRobot, OpenVLA tooling, openpi, and SmolVLA reduce custom implementation into maintained workflows.
Figure 34.7 gives this page a compact map of the interface. Read it left to right, then check whether the surrounding prose names the same observation, action, and evidence contract.
Learning Objectives
- Define a VLA policy in terms of observations, language, proprioception, action chunks, and controllers.
- Explain the lineage from RT-1 to RT-2 and RT-X, including cross-embodiment data.
- Compare Octo, OpenVLA, SmolVLA, RDT-1B, pi-zero, FAST, GR00T, and Gemini Robotics by action representation and evaluation evidence.
- Choose among action tokenization, action chunking, diffusion heads, and flow heads for a robot task.
- Design construct-matched evaluations for VLA behavior, including limitations and safety caveats.
Prerequisites
You should be comfortable with the agent-environment interface, imitation learning, action chunking and diffusion policies, and vision-language models for embodiment. The chapter recaps the needed pieces where they become load-bearing.
Chapter Roadmap
- 34.1 From VLMs to VLAs: the core ideaDefines the VLA interface contract by separating semantic scene understanding from executable robot action generation.
- 34.2 The lineage: RT-1, RT-2, RT-X / Open X-EmbodimentTraces how large robot datasets, action tokenization, and cross-embodiment training moved VLAs from isolated demos to reusable policy families.
- 34.3 Open generalist policies: Octo, OpenVLACompares two influential open routes to generalist robot behavior: diffusion-policy pretraining and open autoregressive VLA backbones.
- 34.4 Diffusion and flow VLAs: RDT-1B, pi-zero, pi-zero FAST, pi-zero point fiveExplains why continuous generative action heads became competitive for dexterous, high-rate, and multi-modal robot trajectories.
- 34.5 Action tokenization vs. continuous heads; the FAST tokenizerStudies the representation trade-off among naive tokens, FAST-style compressed tokens, chunks, and continuous generative heads.
- 34.6 Co-training with web data for semantic generalizationExamines when web-scale semantics really help robot behavior and when embodiment mismatch turns co-training into a misleading shortcut.
- 34.7 Prompting and conditioning embodied policiesShows how prompts, embodiment metadata, and policy-side conditioning signals steer a VLA without obscuring its action contract.
- 34.8 Evaluating VLA behavior; limitations and open problemsBuilds closed-loop, construct-matched evaluation panels that keep per-task, per-embodiment, and safety slices visible.
- 34.9 Action Representations In VLA SystemsTreats the action head as the robot-facing API of a VLA, with explicit trade-offs among tokens, chunks, diffusion, flow, and hierarchical skills.
If you are building, read Sections 34.1, 34.3, 34.5, and the lab in 34.7 first. If you are researching, read the full lineage and evaluation sections, then carry the open questions into Chapter 35.
Hands-On Lab Preview: VLA Adaptation Plan
Objective
Section 34.7 contains the full lab: build a VLA dataset card, prompt set, evaluation panel, and LeRobot or SmolVLA fine-tuning plan.
What You'll Practice
- VLA interface design.
- Prompt conditioning.
- Construct-matched evaluation.
- Open-toolchain planning.
42-Agent Production Checklist Applied
This chapter has been checked against the production team dimensions: chapter scope, curriculum alignment, deep explanation, teaching flow, student questions, cognitive load, examples, exercises, code pedagogy, visual learning, misconceptions, fact integrity, terminology, cross-references, narrative continuity, style, engagement, senior editorial quality, research frontier, structure, content currency, self-containment, opening hook, project work, aha moments, visual identity, demos, memorability, skeptical-reader challenge, prose clarity, pacing, illustrations, epigraph, application examples, fun notes, bibliography, meta-review, controller checks, publication QA, figure fact checking, code captions, and lab design.
For Vision-Language-Action Models, the practical gate is simple: every claim that reaches the chapter body must help a reader build or evaluate an embodied system, and every comparison must be backed by one construct-matched artifact.
Before leaving this chapter, choose one section and name its hook, core mechanism, runnable artifact, figure, misconception warning, exercise, bibliography trail, and evaluation caveat. This quick audit mirrors the 42-agent checklist used for Part VII.
What's Next?
Start with Section 34.1, where we turn the phrase Vision-Language-Action into a precise policy interface.
Brohan et al. (2022). "RT-1: Robotics Transformer for Real-World Control at Scale." arXiv.
RT-1 showed that a transformer policy trained on large real robot data could produce discretized low-level robot actions from images and instructions. It is the starting point for the chapter lineage and useful for readers who want the engineering details behind large-scale robot data collection.
RT-2 made the action-as-language move explicit by fine-tuning VLM backbones to emit robot actions as tokens. Researchers should read it for the co-training setup, while practitioners should read it for the limits of transferring web semantics into motor control.
This paper introduced the cross-institution robot data mixture and RT-X models. It is essential for understanding why embodiment metadata, action normalization, and dataset mixture design matter.
Octo Model Team et al. (2024). "Octo: An Open-Source Generalist Robot Policy." arXiv.
Octo is a transformer-based diffusion policy pretrained on Open X-Embodiment trajectories and designed for flexible fine-tuning. It is the clearest open reference for generalist policy initialization before the Internet-pretrained VLA wave.
Kim et al. (2024). "OpenVLA: An Open-Source Vision-Language-Action Model." arXiv.
OpenVLA connects open VLM backbones to robot action generation and provides a practical codebase for fine-tuning. Practitioners should read it alongside the GitHub repository before adapting an open VLA to a new robot.
Liu et al. (2024). "RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation." arXiv.
RDT-1B studies diffusion transformers for language-conditioned bimanual manipulation at large scale. It is especially relevant for readers comparing tokenized autoregression with continuous denoising heads.
pi-zero uses a flow-matching action head on top of a pretrained vision-language backbone. The paper is central for understanding why continuous action generation became a serious alternative to discretized action tokens.
FAST uses frequency-space compression to tokenize continuous action sequences for autoregressive VLAs. It is the key source for the chapter distinction between naive per-dimension binning and compressed action-sequence tokenization.
Pi-zero point five extends pi-zero through heterogeneous co-training for broader open-world generalization. It is useful for readers studying the frontier between task-specific robot policies and household-scale generalist behavior.
Hugging Face. "LeRobot." GitHub.
LeRobot is the practical open-source toolkit used here for datasets, policy training, evaluation, and low-cost robot workflows. Engineers should start here before writing custom data loaders or training loops.