A robot benchmark is only serious when it can disappoint your favorite model in a way you can replay.
A Skeptical Evaluation Engineer
Chapter 52 turns evaluation from a leaderboard ritual into a scientific instrument. The central object is a matched rollout panel that produces task success, efficiency, safety, robustness, and replay artifacts in one pass.
Evaluation numbers are only comparable when they come from the same task panel, the same metric script, the same hardware or simulator contract, and the same perturbation schedule.
Chapter Overview
This chapter defines the evaluation unit for embodied AI: the closed-loop episode with fixed task sampling, logged perturbations, synchronized monitors, and reproducible aggregation. We work from single-task rollouts to chapter-level benchmark design.
The theory thread covers construct-matched metrics, confidence intervals, statistical validity, and sim-as-proxy arguments. The practical thread shows how to organize logs, replay artifacts, and benchmark manifests so other teams can reproduce claims instead of reinterpreting them.
This chapter keeps a research-grade standard throughout: every promoted claim should be tied to one matched panel, one artifact bundle, and one replay path that lets another team inspect what changed in the closed loop.
Prerequisites
Readers should already know the perception-action loop, simulator workflow, and the role of runtime monitors from earlier parts. Chapter 12 on task suites, Chapter 20 on sim-to-real transfer, and Chapter 51 on lifelong adaptation are especially relevant.
Chapter Roadmap
- 52.1 Why accuracy is not enoughReplace static model scores with closed-loop utility and failure-aware evaluation.
- 52.2 Success rate, path efficiency, time and energy costBuild multi-objective metrics that respect physical resources and timing budgets.
- 52.3 Safety violations and constraint satisfactionMeasure whether the robot stayed inside the operational envelope, not only whether it finished.
- 52.4 Robustness and generalization metricsSeparate nominal performance from perturbation response, OOD behavior, and worst-case tails.
- 52.5 Reproducible evaluation: SIMPLER and sim-as-proxyUse simulator proxies without losing contact with real-world evidence.
- 52.6 Real-world evaluation hygiene; benchmark designPre-register protocols, control operator effects, and audit every comparison artifact.
Use lightweight data classes to make the metric contract explicit, then graduate to DVC, MLflow, Weights and Biases Artifacts, ROS 2 bags, and benchmark manifests. The right tool matters because evaluation work fails more often from missing provenance than from missing model capacity.
The chapter's practical standard is simple: use tools that preserve provenance, timestamps, intervention traces, and replay links. A shorter script is only an advantage when the evidence chain stays intact.
Hands-On Lab: Build the Evaluation Stack
Objective
Build a matched evaluation panel for one embodied task, with one simulator run set and one small physical or replay-based validation set. Save all scalar metrics, rollout traces, perturbation labels, and postmortems in one manifest.
Steps
- Define the task suite, perturbation schedule, and failure taxonomy before choosing a model variant.
- Implement one metric script that computes success, efficiency, safety, and robustness from the same episode table.
- Run a nominal panel, a perturbed panel, and one replay inspection pass.
- Compute confidence intervals or bootstrap bands for each metric.
- Write a one-page review note explaining whether the simulator panel is a usable proxy for the real task.
What's Next?
Continue with Section 52.1: Why accuracy is not enough, where closed-loop utility replaces static accuracy as the central evaluation object.
Read this chapter with the mindset of a reviewer who asks, for every claimed improvement, which panel produced the number, how the perturbations were sampled, and which replay artifact could reproduce the conclusion.
A strong evaluation chapter leaves behind a complete chain: task distribution, environment version, reset policy, seed schedule, metric script, synchronized logs, failure taxonomy, and a result table whose rows can be traced back to episode artifacts.
When reading or teaching the chapter, insist on one more question after every result: which files would another researcher need in order to reproduce, challenge, or extend this exact conclusion without guessing hidden protocol details?
| Tool or Library | Where It Pays Off |
|---|---|
| DVC | Version task manifests, panel definitions, and replay artifacts so benchmark changes are explicit. |
| MLflow or Weights and Biases Artifacts | Store configuration, metric tables, videos, and model checkpoints under one run id. |
| ROS 2 bagging | Capture synchronized sensor, action, and monitor traces for physical-system replay. |
| Pandas plus SciPy | Compute confidence intervals, paired tests, and bootstrap summaries from one episode table. |
| Hydra or OmegaConf | Freeze evaluation configuration so panel drift is visible. |
Extend the lab by adding a second policy family and reusing the exact same panel. If the comparison still needs hand explanations, the protocol is underspecified.
Teach these sections as a sequence from scalar metrics to benchmark governance. Students often understand success rate before they understand why evaluation panels must be matched by construction.
For project-based teaching, insist that every team submit one result artifact with both wins and failure labels. That single discipline turns demonstrations into cumulative evidence.
Each chapter in this part should end with a dossier, not only a plot: configuration, panel definition, metric script, synchronized logs, replay artifact, failure taxonomy, and a short statement of residual uncertainty or residual risk.
A strong seminar or design review should ask four questions at the chapter boundary: what exactly was frozen, what evidence would falsify the claim, which tool preserves the audit trail, and which residual risk or uncertainty still remains after the best current mitigation is applied.
Before leaving the chapter, the reader should be able to design a matched evaluation panel, compute at least one uncertainty interval, explain why a simulator proxy is or is not trustworthy, and audit an invalid comparison.
The chapter succeeds when evaluation becomes an engineering object: matched panels, one metric script, explicit perturbations, and replayable evidence.
Bibliography & Further Reading
Foundational Papers, Tools, and References
Henderson, P. et al. "Deep Reinforcement Learning that Matters." (2018). https://arxiv.org/abs/1709.06560
A classic warning that reported RL gains often collapse under weak evaluation practice.
Agarwal, R. et al. "Deep Reinforcement Learning at the Edge of the Statistical Precipice." (2021). https://arxiv.org/abs/2108.13264
Useful for confidence intervals, aggregate statistics, and matched comparisons.
Brohan, A. et al. "RT-1: Robotics Transformer for real-world control at scale." (2022). https://arxiv.org/abs/2212.06817
A reference point for large-scale real-robot evaluation with task diversity.
Official SIMPLER resources and related sim-to-real benchmark artifacts.
Use the project assets for proxy-evaluation design and traceability across simulation and real execution.