Section 35.2: Cross-embodiment training and transfer

"A trajectory is not transferable until every hidden unit, frame, and control rate has been made painfully explicit."

An Embodiment Auditor

The Real Problem Is Not Data Volume, It Is Data Meaning

Pooling demonstrations from many robots only helps if the shared learner sees comparable events. A mixed dataset can silently combine different camera frames, different control frequencies, different action saturations, and different success definitions. When that happens, the model spends capacity learning translation noise instead of task structure.

This is why cross-embodiment training lives at the boundary between datasets and policy design. The data schema is part of the model architecture. If the schema is wrong, bigger models simply memorize the wrong equivalences more efficiently.

A Canonical Latent Interface

A common pattern is to map robot-specific observations and actions into a canonical latent contract:

$$z_t = E_\text{obs}(o_t, m_r), \qquad a_t^{\text{can}} = N_r(a_t), \qquad \hat a_t = A_r^{-1}(\pi_\theta(z_t, q_t)).$$

Here $m_r$ is embodiment metadata, $N_r$ normalizes robot-specific actions into a canonical representation, and $A_r^{-1}$ maps the canonical action back into robot-specific commands. The policy $\pi_\theta$ only works if those adapters preserve the semantics that matter for transfer.

Code Fragment 1 shows the smallest useful version of that normalization idea. The point is not numerical sophistication. The point is to make the unit conversion visible.

# Normalize position and gripper commands from two robots into one canonical range.
# The policy can only share data when these conventions are explicit.
robots = {
    "arm_a": {"xyz_scale_cm": 1.0, "gripper_closed": 0.0},
    "arm_b": {"xyz_scale_cm": 2.5, "gripper_closed": -1.0},
}

def normalize_command(robot_name: str, dx_cm: float, gripper_value: float) -> tuple[float, float]:
    meta = robots[robot_name]
    canonical_dx = dx_cm / meta["xyz_scale_cm"]
    canonical_gripper = 1.0 if gripper_value == meta["gripper_closed"] else 0.0
    return canonical_dx, canonical_gripper

print(normalize_command("arm_a", dx_cm=1.0, gripper_value=0.0))
print(normalize_command("arm_b", dx_cm=2.5, gripper_value=-1.0))

(1.0, 1.0)
(1.0, 1.0)

The expected output is matching canonical commands for two different robots after embodiment-specific decoding and normalization. If these tuples diverged for the same semantic command, pooled training would quietly mix incompatible actions and poison cross-embodiment transfer.

Where Transfer Usually Breaks

The table above is why cross-embodiment papers talk so much about metadata. Transfer is usually lost at the interface, not at the optimizer.