Section 12.1: Why standardized benchmarks matter

A Careful Control Loop

What This Section Builds

This section makes standardized benchmarks operational. It separates the construct being measured, such as dexterous manipulation or social navigation, from the harness that measures it, such as ManiSkill, robosuite, Habitat, Brax, or Isaac Lab.

The goal is a reproducible habit: define the task contract, freeze the evaluation split, run all methods through the same harness, and save one artifact that contains the configuration and every compared metric.

Theory

We can view a benchmark as a sampled set of closed-loop episodes. For episode $i$ and seed $s$, the harness samples an initial state, exposes observations $o_t$, accepts actions $a_t$, and computes a score $m_i$. The published number is usually an aggregate such as $\frac{1}{N}\sum_i m_i$, so the meaning of the aggregate depends on which episodes entered the sum.

The practical design rule is to make the sampling contract explicit. A claim about general manipulation needs held-out objects, poses, scenes, or task families. A claim about control robustness needs held-out disturbances or physics parameters. A claim about benchmark speed, common in Brax and Isaac Lab comparisons, needs the same number of environments, observation modalities, rollout horizon, accelerator, and logging overhead.

Worked Example

Code Fragment 1 turns the same-config rule into a small audit. The point is not the dataclass itself, it is the habit of refusing to compare two rows unless the panel, split, seeds, wrappers, and metric match exactly.

# Check whether two benchmark rows belong in one comparison table.
# Paper-facing comparisons require the same panel, split, seed set,
# wrapper stack, and metric for every method being compared.
from dataclasses import dataclass

@dataclass(frozen=True)
class EvaluationRun:
    method: str
    panel: str
    split: str
    seed_set: tuple[int, ...]
    wrappers: tuple[str, ...]
    metric: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

baseline = EvaluationRun("BC", "pick-place-v1", "heldout_objects", (0, 1, 2), ("rgbd",), "success_rate")
candidate = EvaluationRun("DiffusionPolicy", "pick-place-v1", "heldout_objects", (0, 1, 2), ("rgbd",), "success_rate")

comparable = baseline.__dict__ | {"method": candidate.method}
same_config = comparable == candidate.__dict__
print(f"paper_table_ready={same_config}")

Practical Recipe

1. Name the construct: manipulation success, lifelong transfer, navigation efficiency, social compliance, or physics throughput.
2. Freeze the task panel, train/validation/test split, seed list, wrapper stack, simulator version, and metric before model selection.
3. Run a transparent baseline through the exact same evaluation script as the proposed method.
4. Report aggregate metrics with confidence intervals or seed-level values, not only a single best run.
5. Record failures as structured cases: perception, planning, contact dynamics, timing, language grounding, human interaction, or evaluation leakage.

Standardized benchmarks matter because embodied AI claims are easy to inflate accidentally. A manipulation policy can memorize training objects, a navigation agent can tune to public validation houses, and a simulator-speed result can change when rendering or logging is included. The benchmark contract prevents these mistakes by specifying what variation is allowed during training and what variation is reserved for evaluation.

The graduate-level habit is to separate three claims. The construct claim says what capability is measured. The harness claim says exactly how episodes are sampled and scored. The evidence claim says which same-config artifact supports the comparison. If those claims are mixed, a paper can appear to compare methods while actually comparing datasets, wrappers, or simulator settings.

A robust benchmark implementation starts with a manifest, not a model. The manifest says which episodes will run, which seeds instantiate them, which wrappers transform observations and actions, and which metric script produces the final table. The same manifest must evaluate the baseline and the proposed method.

1. Write a manifest with task panel, split, seeds, simulator build, wrappers, policy checkpoint, and metric function.
2. Run one deterministic smoke episode and verify that the saved trace matches the manifest.
3. Run every method through one evaluation script, with no method-specific termination or reward shaping.
4. Save per-seed metrics, aggregate metrics, videos or traces, and structured failure labels in one artifact.
5. Promote only same-config comparisons to the paper, keeping exploratory mismatches in diagnostics.

Code Fragment 2 shows the manifest shape that later sections reuse. A JSON file with this schema is more valuable than a screenshot because it makes the evaluation replayable.

# Build a replayable benchmark manifest before training or tuning.
# The manifest captures the evidence boundary that every compared method
# must share for the result to support a paper-facing claim.
from dataclasses import dataclass, asdict

@dataclass
class BenchmarkManifest:
    task_panel: str
    split: str
    seeds: tuple[int, ...]
    simulator: str
    wrappers: tuple[str, ...]
    metric: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

manifest = BenchmarkManifest(
    task_panel="pick-place-v1",
    split="heldout_objects",
    seeds=(0, 1, 2, 3, 4),
    simulator="ManiSkill3",
    wrappers=("rgbd_observation", "dense_action_normalization"),
    metric="success_rate",
)
print(manifest.as_row())

Expected output: the printed manifest should expose the task panel, split, seed policy, simulator, wrapper stack, and metric. If one of those fields is missing, the result is not yet an evaluation artifact.

When a benchmark comparison fails, first ask whether the method failed or the measurement failed. Check for train/test overlap, seed tuning, wrapper drift, simulator-version drift, metric changes, and hidden data augmentation before assigning the result to model quality. This pattern turns a surprising leaderboard row into a reusable diagnostic asset.