Section 23.4: Immersive/VR teleoperation (Open-TeleVision)

"The operator wanted depth perception. The robot delivered a network delay with excellent lighting."

A Stereoscopic Operator

Perception Loop

Teleoperation is a closed-loop control problem with a human inside the loop. The operator observes a rendered state $ ilde{o}_t$, chooses a command $u_t$, and receives delayed feedback $ ilde{o}_{t+\Delta}$. If stereoscopic feedback reduces pose uncertainty but increases delay, the system must quantify the tradeoff.

Useful immersive systems therefore log not only robot actions, but also headset pose, rendered camera stream, frame drops, hand tracking confidence, and command timestamps.

The audit below treats frame drops and delay as episode labels. This is not a performance nicety; it determines whether the resulting demonstration should be considered expert data.

# Classify immersive teleoperation episodes using delay and visual stability.
# The labels help separate expert intent from interface-induced mistakes.
episodes = [
    {"id": "vr001", "delay_ms": 72, "dropped_frames": 1, "tracking": 0.98},
    {"id": "vr002", "delay_ms": 180, "dropped_frames": 9, "tracking": 0.81},
]

for episode in episodes:
    stable_video = episode["dropped_frames"] <= 3 and episode["tracking"] >= 0.95
    acceptable_delay = episode["delay_ms"] <= 100
    label = "clean" if stable_video and acceptable_delay else "interface-risk"
    print(episode["id"], label)

The expected output routes vr002 away from clean training because multiple interface signals are weak at once. This is the central engineering point: immersive teleoperation data should preserve the quality of the operator's perceptual channel. If a policy later fails on an episode labeled interface-risk, the team should inspect video delay, dropped frames, and hand tracking before blaming the manipulation model.

Mechanism: Active Visual Feedback

Open-TeleVision-style systems change the information available to the operator. A fixed camera gives a passive view of the scene; active stereo feedback lets the operator move viewpoint, resolve occlusions, and align the robot body with task-relevant geometry. That extra perception can improve demonstrations for tasks such as opening drawers, threading tools, or reaching around clutter, where the critical state is not visible from one fixed camera.

The cost is that active feedback adds a second policy-like behavior: the human chooses not only hand motion, but also where to look. A dataset that keeps headset pose and camera motion can later train or evaluate active perception policies. A dataset that stores only wrist images discards the reason immersion helped.

Concrete Tool Anchors

A practical immersive stack usually combines several maintained layers. Open-TeleVision provides a reference design for immersive active visual feedback and imitation-learning data collection. ROS 2 or a similar robot middleware layer carries robot state, commands, emergency-stop state, and camera topics. WebRTC-style streaming or vendor headset SDKs carry the visual channel. A calibration tool such as Kalibr, OpenCV calibration routines, or the platform's own stereo calibration workflow records the geometry that makes depth perception meaningful.

The important engineering decision is where each timestamp is created. Camera capture time, network receive time, headset render time, hand-tracking time, and robot command time should all be recorded or reconstructable. Without those anchors, a replay can look smooth while hiding the delay that shaped the operator's actions.