Section 12.2: Manipulation: ManiSkill3, robosuite, RoboCasa, robomimic, RLBench

A Careful Control Loop

What This Section Builds

This section makes manipulation benchmarks operational. It teaches how to read a tabletop result as a claim about contact, vision, demonstration learning, object generalization, task generalization, or scene diversity.

The goal is not to memorize benchmark names. The goal is to design a comparison where ManiSkill3 speed, robosuite controller choices, RoboCasa scene diversity, robomimic datasets, and RLBench task variation are treated as evidence boundaries rather than interchangeable labels.

Theory

A manipulation benchmark samples an initial object arrangement, exposes robot observations, accepts actions through a controller, and scores whether task predicates become true. The same task name can change meaning when the action space changes from end-effector deltas to joint torques, when RGB observations become privileged state, or when demonstrations come from a different controller.

The split must match the claim. For imitation learning, separate training and test demonstrations so trajectory memorization is not counted as skill. For object generalization, hold out object instances or categories. For task generalization, hold out task templates, not merely random seeds for the same template.

Worked Example

Code Fragment 1 shows a compact leakage audit for manipulation. If the training and test panels share object identifiers, task templates, or demonstration identifiers, the reported success rate may reflect familiarity rather than manipulation competence.

# Audit manipulation splits for object, task, and demonstration leakage.
# The result should be empty before a manipulation benchmark supports
# a generalization claim about held-out episodes.
from dataclasses import dataclass

@dataclass(frozen=True)
class ManipulationEpisode:
    task: str
    object_id: str
    demo_id: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

train = {
    ManipulationEpisode("lift", "mug_001", "demo_017"),
    ManipulationEpisode("stack", "cube_002", "demo_021"),
}
test = {
    ManipulationEpisode("lift", "mug_009", "demo_110"),
    ManipulationEpisode("stack", "cube_002", "demo_222"),
}

train_objects = {episode.object_id for episode in train}
test_objects = {episode.object_id for episode in test}
print(f"object_leakage={sorted(train_objects & test_objects)}")

Practical Recipe

1. Choose the suite that matches the claim: demonstrations for robomimic, task variety for RLBench, controller studies for robosuite, GPU-parallel training for ManiSkill3, or household-style manipulation for RoboCasa.
2. Freeze train, validation, and test splits by task template, object instance, scene, and demonstration identifier.
3. Use the same observation keys, action space, controller, horizon, and success predicate for every method in the comparison.
4. Report per-task and per-seed success, not only a mean that hides brittle tasks.
5. Label failures as perception miss, grasp/contact failure, controller saturation, task-predicate miss, recovery failure, or split leakage.

Manipulation benchmarks become useful when they make hidden choices visible. A policy trained on robomimic demonstrations may be excellent at reproducing dataset actions but weak under new object placements. A policy trained in ManiSkill3 may benefit from high-throughput exploration but still need an object and task split that tests transfer. A robosuite controller comparison may be valid only inside the chosen action interface.

The graduate-level habit is to ask what each suite contributes to the evidence. ManiSkill3 and Isaac-style GPU workflows help scale rollout counts, robosuite makes controller and robot-composition choices explicit, RoboCasa and BEHAVIOR-style assets broaden household variation, robomimic defines offline demonstration evaluation, and RLBench tests task-level variation with demonstrations. The paper claim should name which of those dimensions it actually measured.

A robust manipulation benchmark starts with a split file. The split file should list task templates, object IDs, scene IDs, demonstration IDs, and seeds. The evaluation runner should consume that file for every method so a new policy cannot quietly choose easier episodes.

1. Choose one primary suite and write the task contract in that suite's native terms.
2. Freeze task, object, scene, demonstration, and seed splits before hyperparameter tuning.
3. Evaluate baselines and candidates with the same observation keys, action space, controller, and horizon.
4. Save per-episode success, reward, horizon length, contact or grasp status, and video trace.
5. Aggregate by task family and seed so one easy task cannot dominate the conclusion.

Code Fragment 2 turns the manipulation protocol into a result artifact. The important field is split_name, because it keeps a reported success rate attached to the held-out condition it actually measured.

# Record a manipulation result with the split and controller attached.
# A success rate without these fields is not comparable across suites
# or even across two runs from the same suite.
from dataclasses import dataclass, asdict

@dataclass
class ManipulationResult:
    suite: str
    split_name: str
    controller: str
    observation: str
    seeds: tuple[int, ...]
    success_rate: float

    def as_row(self) -> dict[str, object]:
        return asdict(self)

result = ManipulationResult(
    suite="RLBench",
    split_name="heldout_task_variations",
    controller="end_effector_delta_pose",
    observation="front_rgb+proprioception",
    seeds=(10, 11, 12, 13, 14),
    success_rate=0.62,
)
print(result.as_row())

When a manipulation experiment fails, localize the failure before changing the model. Replay the episode and tag whether the failure came from perception, grasp approach, contact instability, controller saturation, recovery, task-predicate scoring, or a held-out split mismatch. This prevents a model change from masking an evaluation problem.