Section 34.8: Evaluating VLA behavior; limitations and open problems

"A robot policy is a promise about the next second of the world."

A Grounded AI Agent

Figure 34.8 gives this page a compact map of the interface. Read it left to right, then check whether the surrounding prose names the same observation, action, and evidence contract.

Evaluation Must Be Closed-Loop

A VLA does not earn trust by producing plausible action tokens on a held-out dataset. It earns trust by acting in a closed loop under fixed, inspectable conditions. The basic metrics are task success, safety violations, time, energy, intervention count, recovery rate, and latency. The deeper question is whether those metrics were co-computed on the same panel of episodes.

Code Fragment 1 gives the evaluation habit this chapter wants: compare policies on the same episodes, with the same seeds, and with all metrics computed in one pass.

# Co-compute success, safety, and latency metrics on one shared evaluation panel.
# This prevents invalid comparisons across different tasks, seeds, or robot setups.
import numpy as np

episodes = np.array([
    [1, 0, 180],
    [1, 0, 195],
    [0, 1, 240],
    [1, 0, 210],
])
success = episodes[:, 0].mean()
safety_violation = episodes[:, 1].mean()
latency_ms = episodes[:, 2].mean()
print(f"success={success:.2f}")
print(f"safety_violation={safety_violation:.2f}")
print(f"latency_ms={latency_ms:.1f}")

success=0.75
safety_violation=0.25
latency_ms=206.2

Limitations

VLA systems remain brittle in ways that matter for deployment. They can overfit to data collection viewpoints, confuse similar objects, miss small contact events, issue actions outside safe limits, and fail under distribution shift. Their language capability can also create false confidence: a fluent explanation of a failed action does not make the action safe.

Evaluation Checklist

1. Freeze the robot, task, cameras, controller, prompts, and evaluation seeds.
2. Run all policy variants on the same episode panel.
3. Compute success, safety, latency, interventions, and recovery from the same logs.
4. Save failure videos and classify errors by perception, grounding, action representation, control, or evaluation.
5. Report vendor claims separately from independently reproduced results.

Open Problems

The field still lacks reliable answers to several questions. How should action representations transfer across embodiments? Which data mixtures improve real tasks rather than benchmark scores? How can a VLA know when not to act? How should long-horizon planners hand tasks to low-level VLAs? How do we certify safety when the policy is a large generative model?

Expected output: Evaluating VLA behavior; limitations and open problems should leave a reproducible VLA evidence trace with checkpoint, action representation, robot interface, metric, and failure label.