Section 23.2: Leader-follower teleoperation (ALOHA, GELLO)

"The best joystick for an arm is sometimes another arm that admits it is only pretending."

A Kinematic Twin

Kinematic Mapping

The cleanest leader-follower interface makes the leader configuration $q_L$ correspond to a follower configuration $q_F$. The simplest mapping is an affine calibration in joint space:

$$q_F = S(q_L - q_L^0) + q_F^0,$$

where $q_L^0$ and $q_F^0$ are neutral poses and $S$ contains joint sign and scale terms. More complex systems map end-effector poses through forward and inverse kinematics, but the principle is the same: the operator should not mentally solve geometry that the device can embody.

Latency And Stability

Latency changes what the operator sees and what the robot executes. Let $\Delta t = t_{execute} - t_{observe}$. At 50 Hz, a control step is 20 ms; at 5 Hz, it is 200 ms. For contact-rich manipulation, that difference can decide whether the operator corrects a slip or records an avoidable failure.

The following snippet computes a latency budget and flags episodes that should not be used as clean expert data.

# Audit leader-follower timing so bad labels are not treated as expert actions.
# Episodes over the latency threshold should be labeled or excluded from clean splits.
episodes = [
    {"id": "ep001", "control_hz": 50, "network_ms": 18, "camera_ms": 12},
    {"id": "ep002", "control_hz": 10, "network_ms": 55, "camera_ms": 40},
]

for episode in episodes:
    control_ms = 1000 / episode["control_hz"]
    total_ms = control_ms + episode["network_ms"] + episode["camera_ms"]
    label = "clean" if total_ms <= 80 else "latency-risk"
    print(episode["id"], round(total_ms, 1), "ms", label)

The expected output distinguishes an episode that can be treated as clean supervision from one that needs a latency-risk label. That distinction matters because a delayed correction can look like a bad action in the dataset even when the operator made the right decision based on stale feedback. A serious collection pipeline keeps both rows, but routes them to different training or stress-evaluation uses.