Section 24.3: Cross-embodiment pooling

"The robot arms agreed to share data. Their grippers asked to see the contract."

A Diplomat For Robot Bodies

Normalization Targets

Pooling can happen in observation space, action space, latent space, or task space. A common strategy is to express actions as end-effector deltas, normalize continuous values, and provide embodiment tokens or metadata so the model knows which body produced the trajectory.

For an action $a_t$ with dataset mean $\mu_e$ and scale $\sigma_e$ for embodiment $e$, a normalized action can be written as:

$$ ilde{a}_t = (a_t - \mu_e) / \sigma_e.$$

The subscript matters. Using a global mean across incompatible robots can hide systematic body differences and produce commands that are invalid for smaller or slower platforms.

Code Fragment 1 demonstrates embodiment-aware normalization for two robots with different action ranges.

# Normalize actions per embodiment so robot-specific scales are preserved.
# The model can then receive both the normalized action and the embodiment id.
actions = {
    "small_arm": [0.01, 0.02, 0.03],
    "mobile_dual_arm": [0.08, 0.10, 0.12],
}

for robot, values in actions.items():
    mean = sum(values) / len(values)
    scale = max(values) - min(values)
    normalized = [round((v - mean) / scale, 2) for v in values]
    print(robot, normalized)

The expected output shows identical normalized values for two robots, but the interpretation is not "the robots did the same thing." It means each robot's local action range has been centered and scaled. A training batch should therefore keep the normalized action together with an embodiment identifier, action-unit metadata, and the inverse transform needed to recover a valid hardware command.

Mechanisms For Cross-Embodiment Transfer

Cross-embodiment transfer usually relies on one of three mechanisms. The first is shared task semantics: language or goal images say "pick up the mug" even when two robots use different joints. The second is shared observation structure: cameras, object states, and scene geometry can be encoded by a common visual backbone. The third is conditioned action decoding: the policy predicts actions in a representation that is decoded differently for each robot body.

These mechanisms make different assumptions. Shared task semantics assumes that labels are consistent across datasets. Shared observation structure assumes that visual features relevant to one robot also matter to another. Conditioned action decoding assumes the model receives enough morphology and control-mode metadata to avoid producing commands that the target robot cannot execute.

Toolchain Pattern

A practical cross-embodiment stack starts with source-specific loaders, converts each source into a common episode schema, then adds an embodiment adapter. The adapter can be as simple as a robot-id embedding and per-robot action normalizer, or as complex as a morphology graph encoding links, joints, limits, and gripper type. The important rule is that adapter inputs are saved in the dataset card, not hidden in model code.