Section 38.6: World models for visual control

"In hardware, the rejection decision is often more important than the nominal prediction."

A Latent State That Must Survive Contact

Problem First

A world model that looks great on benchmark rollouts can still fail the moment vision becomes ambiguous, latency spikes, or the robot makes contact. Visual control is the acid test because the learned state must handle high-dimensional sensing while preserving the low-dimensional geometry and timing that control depends on.

Core Model

A deployed visual-control latent often combines multiple sensing streams: $$z_t = f_\theta\big(\mathrm{enc}_{\text{vision}}(o_t^{1:m}), \mathrm{enc}_{\text{prop}}(q_t, \dot q_t), a_{t-1}, h_{t-1}\big).$$ This matters because vision alone rarely resolves hidden contact state, while proprioception alone rarely resolves scene structure.

The control objective remains the same, but rollout error should now be read through a safety lens: $$a_t = \pi(z_t), \qquad \text{reject if } \Pr(\text{collision or instability} \mid z_t) > \tau.$$ In practice, the world model becomes one module in a larger stack that may include a low-level stabilizer, safety filter, or reflex policy.

Visual control therefore favors representations that are task-sufficient, multimodal, and timing-aware. A perfect decoder is optional. Stable contact prediction, fast inference, and recoverable failure handling are not.

Minimal Probe

The probe below fuses a visual embedding with proprioception and then checks whether a simple confidence gate would reject an unsafe rollout. The logic is intentionally small, because this is the boundary every hardware stack eventually needs to expose.

# Fuse vision and proprioception, then gate execution by confidence.
# The rejection decision is often more important than the nominal action.
import numpy as np

vision_latent = np.array([0.62, 0.18, 0.51])
proprio_latent = np.array([0.55, 0.24, 0.48])
fused = 0.7 * vision_latent + 0.3 * proprio_latent
uncertainty = np.abs(vision_latent - proprio_latent).mean()
execute = uncertainty < 0.08
print({"fused_state": np.round(fused, 3).tolist(), "uncertainty": round(float(uncertainty), 3), "execute": execute})

Expected behavior: Execution should proceed only when the sensing streams agree closely enough for the controller to trust the fused state. If uncertainty stays high during contact or occlusion, the stack needs a fallback mode rather than a stronger decoder.

Practical Recipe

1. Log vision-only, proprio-only, and fused-state diagnostics separately.
2. Define a rejection policy for latent uncertainty before the first hardware test.
3. Stress the model with lighting change, occlusion, calibration drift, and mild contact mismatch.
4. Measure wall-clock latency alongside task success; a stronger latent that arrives too late is still a failure.

Visual control is where world models stop being abstract. The useful representation must carry geometry, embodiment, and timing in one state. That often means a multimodal latent, a shorter imagination horizon than benchmark videos suggest, and an explicit contract for when to reject the model's advice.

There is also a design decision about decoding. Some teams keep an image decoder because reconstructions expose what the latent forgot. Others remove it and devote capacity to reward, value, or contact heads. The better choice depends on what failures the builder needs to diagnose and how much inference budget the controller has.