Section 23.3: Handheld and in-the-wild collection (UMI)

"I collected the demonstration in a kitchen. The robot insists it has never been to one."

A Portable Gripper

From Human Motion To Robot Action

The core abstraction is a relative trajectory. Instead of asking the policy to imitate absolute human pose, the dataset can store local end-effector displacements over a short horizon:

$$\Delta x_{t:t+H} = (T_t^{-1}T_{t+1}, T_t^{-1}T_{t+2}, \ldots, T_t^{-1}T_{t+H}),$$

where $T_t$ is the gripper pose at time $t$. Relative actions reduce dependence on a specific room coordinate frame and make it easier to deploy the learned behavior on a robot with a different base pose.

Code Fragment 1 shows a compact version of the relative-action transform used in many trajectory datasets. The example uses one-dimensional poses so the arithmetic is inspectable, but the same idea extends to full SE(3) transforms from Chapter 4.

# Convert absolute gripper positions into local relative actions.
# This teaches the representation before a full SE(3) trajectory library hides it.
positions_m = [0.40, 0.43, 0.47, 0.46]
relative_actions = []

for current, nxt in zip(positions_m, positions_m[1:]):
    relative_actions.append(round(nxt - current, 3))

print(relative_actions)
print("largest step:", max(abs(x) for x in relative_actions), "m")

Data Collection Recipe

1. Record synchronized egocentric video, gripper pose, gripper width, and task instruction.
2. Calibrate gripper geometry and camera extrinsics before each session.
3. Store both absolute sensor poses and relative policy actions.
4. Match inference-time latency during training so the policy sees the same delay it will experience on hardware.
5. Validate with a robot replay protocol before counting the episode as deployable data.

pip install pandas

# Build a handheld-demonstration manifest with one deliberate audit field.
# The example includes the stress condition explicitly so the audit can run end to end.
episodes = [
    {"id": "umi_001", "camera_calibrated": True, "latency_ms": 65, "split": "train"},
    {"id": "umi_002", "camera_calibrated": "latency_review", "latency_ms": 140, "split": "stress"},
]
print(episodes)

# Apply a readiness rule that separates clean data from stress or repair data.
for episode in episodes:
    ready = episode["camera_calibrated"] is True and episode["latency_ms"] <= 100
    episode["split"] = "train" if ready else "review"
summary = {split: sum(ep["split"] == split for ep in episodes) for split in {"train", "review"}}
print(summary)

# Complete manifest audit for handheld demonstrations.
# It keeps clean training data separate from risky but useful stress examples.
episodes = [
    {"id": "umi_001", "camera_calibrated": True, "latency_ms": 65, "split": "train"},
    {"id": "umi_002", "camera_calibrated": False, "latency_ms": 140, "split": "stress"},
]
for episode in episodes:
    ready = episode["camera_calibrated"] is True and episode["latency_ms"] <= 100
    route = "clean-train" if ready else "stress-or-recollect"
    print(episode["id"], route)