Section 9.5: Benchmark environment map

A Careful Control Loop

Benchmarks Are Constructs, Not Leaderboards

A benchmark environment measures a construct: locomotion stability, contact-rich manipulation, multi-agent coordination, navigation under partial observability, household task completion, visual grounding, or sim-to-real robustness. The benchmark's API, assets, physics, sensor model, and metric define that construct.

The practical question is not which environment is most popular. The question is which environment makes the right failure possible. A grasping benchmark that never models contact slip cannot validate contact robustness. A household benchmark without persistent object state cannot validate long-horizon interaction.

Environment Families And Valid Claims

Gymnasium-style APIs are useful for clean reinforcement-learning contracts. PettingZoo extends that discipline to multi-agent settings. MuJoCo, robosuite, ManiSkill, and Isaac Lab are often chosen for contact, manipulation, locomotion, vectorized rollouts, and controller integration. Habitat, AI2-THOR, ProcTHOR, BEHAVIOR, and OmniGibson emphasize embodied navigation, household semantics, object state, and visual interaction.

Each family has a validity envelope. If the claim is about torque-level transfer, check dynamics, contacts, actuation, and timing. If the claim is about embodied household reasoning, check scene diversity, affordances, persistent state, and task definitions. If the claim is about policy comparison, check whether all methods run through the same wrapper and metric script.

Worked Example

Code Fragment 9.5.1 scores candidate benchmark families against a task claim. The point is not to rank tools globally. The point is to make the construct match explicit before the experiment starts.

# Match benchmark families to the construct a claim needs.
# Weak matches become limits on what the result can support.
claim_needs = {"contact", "object_state", "vision", "failure_labels"}
benchmarks = {
    "MuJoCo manipulation": {"contact", "controller_timing", "failure_labels"},
    "ManiSkill": {"contact", "vision", "object_state", "failure_labels"},
    "Habitat navigation": {"vision", "layout_diversity", "navigation_metrics"},
    "ProcTHOR household": {"vision", "object_state", "layout_diversity"},
}

for name, supports in benchmarks.items():
    missing = sorted(claim_needs - supports)
    verdict = "candidate" if not missing else f"claim limit: missing {missing}"
    print(f"{name}: {verdict}")

MuJoCo manipulation: claim limit: missing ['object_state', 'vision']
ManiSkill: candidate
Habitat navigation: claim limit: missing ['contact', 'failure_labels', 'object_state']
ProcTHOR household: claim limit: missing ['contact', 'failure_labels']

Practical Recipe

1. Write the claim as a construct: contact robustness, visual navigation, household state tracking, coordination, or transfer.
2. Choose an environment family whose observation, action, asset, reset, and metric contract can measure that construct.
3. Run a random policy, a scripted baseline, and the intended policy through the same wrapper to test logging and metric sanity.
4. Record failures as structured cases: perception, state, planning, control, task semantics, timing, or metric mismatch.
5. Report the benchmark's unsupported assumptions beside the positive result.

Benchmark environment map becomes useful when it is tied to a closed-loop contract. The contract names the observation stream, action representation, reset distribution, timing budget, metric, assets, version, and evaluation artifact. Without that contract, a model can look capable in a benchmark table while failing the first time the real task changes an object state the environment never modeled.

The graduate-level habit is to separate three claims. The benchmark claim says what construct the environment measures. The systems claim says what wrapper and artifact make the result reproducible. The transfer claim says which real-world assumptions remain untested. Keeping those claims separate prevents benchmark convenience from becoming benchmark overreach.

A robust implementation starts with a benchmark card before it starts with a policy. The card should log the environment version, wrapper stack, task panel, observation and action spaces, seeds, metric script, asset set, and failure taxonomy. The policy result is interpretable only after that card exists.

1. Write a benchmark card with construct, environment version, wrapper stack, seeds, metric, and failure labels.
2. Run random and scripted baselines to catch broken resets, hidden privileged state, and metric leakage.
3. Freeze the held-out task panel before tuning policies or curricula.
4. Save videos, traces, configs, metric outputs, and failure labels in one artifact bundle.
5. Compare methods only when one script evaluates them on the same benchmark card.

# Build a benchmark card before reporting a policy score.
# The card states what construct the environment can validly measure.
from dataclasses import dataclass, asdict

@dataclass
class BenchmarkCard:
    environment: str
    construct: str
    observation_space: str
    action_space: str
    metric: str
    known_limit: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

card = BenchmarkCard(
    environment="ManiSkill pick-and-place panel",
    construct="vision-conditioned contact manipulation",
    observation_space="RGB-D camera plus proprioception",
    action_space="end-effector delta pose",
    metric="held-out object success rate with failure labels",
    known_limit="requires real friction calibration before deployment claims",
)
print(card.as_row())

{'environment': 'ManiSkill pick-and-place panel', 'construct': 'vision-conditioned contact manipulation', 'observation_space': 'RGB-D camera plus proprioception', 'action_space': 'end-effector delta pose', 'metric': 'held-out object success rate with failure labels', 'known_limit': 'requires real friction calibration before deployment claims'}

Expected output: the record exposes the benchmark's construct and its known limit before any model result appears. That ordering keeps the environment from being used to support claims it cannot measure.

When a benchmark experiment fails, avoid labeling the method as weak before checking the wrapper and construct. First assign the failure to perception, state estimation, planning, control, task semantics, timing, data coverage, or metric mismatch. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing benchmark score into a reusable diagnostic asset.