Section 40.4: Self-supervised pretraining for control

Pretraining is only worth the electricity bill if the robot needs fewer collisions to learn the same lesson.

A Patient JEPA Encoder

Self-supervised pretraining becomes useful for control only when the frozen or lightly adapted encoder exposes the state variables the controller cannot cheaply relearn from sparse robot rewards. In practice, those variables are often object permanence, coarse geometry, temporal continuity, and action-relevant scene changes.

The engineering question is therefore not "should we pretrain?" but "which objective, transfer interface, and adaptation budget give the controller the highest return per hour of robot data?"

Theory

Let $\phi(o_t)$ be a pretrained encoder and $\pi_\theta(a_t \mid \phi(o_t), g_t)$ a downstream controller conditioned on goal $g_t$. The transfer question is whether pretraining improves the data-efficiency or asymptotic quality of the control objective

$$ J(\theta; \phi)=\mathbb{E}\left[\sum_{t=0}^{T-1}\gamma^t r(s_t,a_t)\right]. $$

In practice the encoder can be frozen, partially adapted, or fully fine-tuned. A convenient decomposition is

$$ \phi^\star=\arg\min_\phi \mathcal{L}_{\text{ssl}}(\phi), \qquad \theta^\star=\arg\max_\theta J(\theta;\phi^\star), $$

followed by a decision about whether joint fine-tuning is worth the extra instability. The control win comes from giving the policy an input representation that already respects the structure of the task before expensive interaction begins.

Worked Example

The following probe compares two control-learning curves under a simple transfer model. The point is to inspect what a pretraining gain looks like numerically before it gets buried in a larger robot stack.

# Compute a JEPA-style representation prediction loss.
# Compare sample-efficiency with and without a pretrained encoder.
import numpy as np

robot_hours = np.array([1, 2, 4, 8, 16], dtype=np.float32)
scratch_success = np.array([0.18, 0.27, 0.39, 0.51, 0.64], dtype=np.float32)
pretrained_success = np.array([0.31, 0.42, 0.55, 0.66, 0.75], dtype=np.float32)

gain = pretrained_success - scratch_success
best_hour = int(robot_hours[np.argmax(gain)])
print({
    "scratch_final": round(float(scratch_success[-1]), 2),
    "pretrained_final": round(float(pretrained_success[-1]), 2),
    "largest_gain_hour": best_hour,
    "largest_gain": round(float(np.max(gain)), 2),
})

The expected output should show the largest gain at an early or middle interaction budget. If the gain only appears after large amounts of robot data, the encoder may be helping optimization less than a better controller or dataset would.

In production, the decisive question is where the representation enters the controller. Frozen latents are attractive because they stabilize training and simplify debugging, adapter-based tuning is often the best compromise when the task differs from pretraining, and full fine-tuning should be reserved for cases where the embodiment mismatch is large enough that a fixed encoder blocks performance.

V-JEPA 2 is a useful anchor because it separates broad passive pretraining from smaller embodiment-specific adaptation. That pattern generalizes beyond JEPA: whenever robot data is expensive, treat pretraining as a way to purchase state abstraction early, then verify the win with matched closed-loop evidence.

1. Write the observation, action, state estimate, success metric, and rejection criterion.
2. Run a deterministic smoke test on one seed and save the complete configuration.
3. Add one perturbation tied to the section topic: delay, noise, horizon length, contact change, distractor object, or generated-scene shift.
4. Compare only methods evaluated by the same script, split, seed panel, and metric definition.
5. Record a postmortem that assigns failures to perception, representation, dynamics, planning, control, data coverage, timing, or evaluation.

When Self-supervised pretraining for control fails, do not collapse the result into a single method verdict. Assign the failure to the interface that broke, rerun one controlled perturbation, and keep the trace next to the metric. That habit turns a disappointing rollout into a reusable diagnostic asset.