Section 24.1: The major datasets: Open X-Embodiment, DROID, BridgeData V2, RH20T, RoboMIND, AgiBot World

"A million trajectories sound large until you ask how many robot bodies they remember."

A Dataset Cartographer

Dataset Families

Open X-Embodiment pools many robots and tasks to study RT-X style cross-robot transfer. DROID emphasizes in-the-wild collection across diverse scenes and data collectors. BridgeData V2 emphasizes large-scale manipulation trajectories with language and goal-conditioning compatibility. RoboNet is historically important because it pushed multi-robot video prediction and self-supervised interaction data before the current foundation-model wave.

The snippet below encodes a small dataset card and computes a crude coverage score. It is not a substitute for careful evaluation, but it forces the reader to separate trajectory count from embodiment and task diversity.

# Compare dataset cards by coverage factors, not only trajectory count.
# The score is a teaching proxy for reading dataset claims critically.
datasets = {
    "Open X-Embodiment": {"robots": 22, "tasks": 527, "scenes": 21},
    "DROID": {"robots": 1, "tasks": 86, "scenes": 564},
    "BridgeData V2": {"robots": 1, "tasks": 24, "scenes": 24},
}

for name, card in datasets.items():
    coverage_proxy = card["robots"] * card["tasks"] * card["scenes"]
    print(name, coverage_proxy)

The expected output ranks Open X-Embodiment highest because the proxy multiplies robots, tasks, and scenes. That does not mean it is best for every research question. It means the dataset card exposes more cross-factor coverage under this particular proxy, while DROID's high scene count may be more relevant for in-the-wild visual generalization and BridgeData V2's task framing may be more relevant for language-conditioned tabletop studies.

Mechanism: What The Dataset Teaches

Each major dataset teaches a policy through a different pressure. Open X-Embodiment pressures the model to find representations that survive changes in robot body and task source. DROID pressures the model to survive scene, collector, and household variation. BridgeData V2 pressures the model to connect manipulation behavior with language or goal-conditioned task descriptions. RoboNet-style video-interaction datasets pressure predictive models to learn how robot actions change visual futures.

This distinction matters when choosing pretraining data. A model trained on many robots may learn robust visual affordances but still need an adapter for a new action space. A model trained in many homes may learn visual robustness but remain tied to one hardware platform. A model trained with rich language may follow instructions better while still failing on contact dynamics that the dataset under-sampled.

Toolchain And Split Advice

A practical researcher should load the official release first, preserve its native split names, then create a project-specific split manifest only after deciding what generalization claim is being tested. For Open X-Embodiment, that may be held-out robot or held-out task. For DROID, it may be held-out scene, collector, or object. For BridgeData V2, it may be held-out instruction, environment, or goal image. A split that ignores the dataset's main source of diversity wastes the reason to use that dataset.