Section 34.1: From VLMs to VLAs: the core idea

"A VLA policy is a contract between language grounding, image tokens, robot state, and action decoding."

A Grounded AI Agent

Why VLMs Are Not Enough

A vision-language model can describe a scene, answer a question, or identify an object. That is useful, but embodiment adds a harder requirement: the answer must change the world. If the instruction is "put the cup on the coaster," the system must decide where the cup is, where the coaster is, which grasp is feasible, how the arm should move, when the gripper should close, and how to recover if the cup slips.

A VLA model extends the multimodal stack by adding an action channel. Formally, it learns a policy $\pi_\theta(a_{t:t+H} \mid o_{1:t}, q, r_t)$, where $o$ is visual observation, $q$ is the language instruction, $r_t$ is robot state, and $a_{t:t+H}$ is an action chunk over a short horizon. The exact action representation varies across systems: discrete tokens in RT-1 and RT-2, diffusion or flow outputs in RDT and pi-zero, and compressed action tokens in FAST-style autoregressive policies.

The Interface Contract

The most useful way to read any VLA paper is to ask four interface questions. What observations enter the model? What robot state is exposed? What action space exits the model? What controller consumes those actions? This contract is the bridge back to Chapter 2 on action representations and Chapter 7 on controllers versus policies.

Code Fragment 1 makes the VLA interface concrete with typed containers. It does not run a neural policy, it shows the contract that every neural policy must satisfy.

# Minimal VLA interface: image features, instruction text, and robot state enter together.
# The policy returns an action chunk, not a single ungrounded language answer.
from dataclasses import dataclass
import numpy as np

@dataclass
class VLAObservation:
    image_embedding: np.ndarray
    instruction: str
    joint_state: np.ndarray

    def as_row(self) -> dict[str, object]:
        return asdict(self)

@dataclass
class ActionChunk:
    delta_xyz: np.ndarray
    gripper_open: np.ndarray

obs = VLAObservation(
    image_embedding=np.array([0.12, 0.88, 0.41]),
    instruction="pick up the red block",
    joint_state=np.array([0.0, 0.4, -0.2, 0.1]),
)
chunk = ActionChunk(
    delta_xyz=np.array([[0.02, 0.00, -0.01], [0.01, 0.01, -0.02]]),
    gripper_open=np.array([1.0, 0.0]),
)
print(obs.instruction)
print(chunk.delta_xyz.shape)

pick up the red block
(2, 3)

# LeRobot shortcut: inspect the observation and action schema before training.
# The dataset object exposes cameras, robot state, language task, and action chunks.
from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

dataset = LeRobotDataset("lerobot/aloha_static_coffee")
print(dataset.features.keys())
print(dataset.meta.info.get("fps"))

Figure 34.1 should be read as the minimal VLA interface: instruction, visual observation, proprioception, action representation, and rollout evidence must all be named before behavior is interpreted.

Failure Modes

A VLA can fail even when its language understanding looks strong. It can hallucinate affordances, issue an action outside the robot workspace, ignore proprioceptive limits, move too slowly for a dynamic scene, or overfit to camera viewpoints seen during data collection. These are not minor implementation details. They are the reasons VLA evaluation must be closed-loop and robot-aware.

Expected output: From VLMs to VLAs: the core idea should leave a reproducible VLA evidence trace with checkpoint, action representation, robot interface, metric, and failure label.