Section 40.3: V-JEPA 2 and action-conditioned latent planning

"A world model becomes actionable the moment you let the next action change the future it predicts."

An Action-Conditioned Predictor

From Passive Prediction To Action-Conditioned Rollouts

Video pretraining alone can tell you which latent futures are plausible, but it cannot tell you which future follows from a chosen action sequence. That distinction matters in robotics. A representation that knows a mug can move is not enough; a planner must know which latent mug motion corresponds to closing the gripper, rotating the wrist, or aborting the reach.

V-JEPA 2 addresses this by keeping the self-supervised video backbone and then post-training an action-conditioned world model, often denoted V-JEPA 2-AC in the paper. The latent transition becomes:

$$ z_{t+1} = h_\phi(z_t, a_t), \qquad \hat z_{t+1:t+H} = h_\phi(z_t, a_{t:t+H-1}) $$

where $z_t$ comes from the pretrained video encoder and the action-conditioned head learns how control inputs move the latent state across a horizon $H$.

Planning Objective

For image-goal planning, the planner searches over candidate action sequences and scores them by how close the predicted terminal latent is to the goal latent:

$$ a_{t:t+H-1}^\star = \arg\min_{a_{t:t+H-1}} \left\lVert h_\phi(z_t, a_{t:t+H-1}) - z_g \right\rVert_2^2 + \lambda C(a_{t:t+H-1}) $$

Here $z_g$ is the goal-image embedding and $C$ is any extra action cost, such as smoothness, collision proxy, or control effort. This looks like model-predictive control in latent space, because that is exactly what it is. The novelty is that the dynamics live in a self-supervised latent representation rather than an analytic state space.

Worked Planning Probe

Code Fragment 40.3.1 demonstrates the core idea with a tiny latent transition model. The point is to make latent-space goal planning concrete before discussing the full V-JEPA 2 system.

# Roll a latent state forward under candidate actions and score
# each plan by terminal distance to a goal embedding.
import numpy as np

z0 = np.array([0.2, -0.1], dtype=np.float32)
zg = np.array([0.9, 0.4], dtype=np.float32)
candidates = [
    np.array([[0.20, 0.10], [0.18, 0.09], [0.15, 0.08]], dtype=np.float32),
    np.array([[0.10, 0.18], [0.11, 0.17], [0.12, 0.16]], dtype=np.float32),
]

def rollout(z, actions):
    for action in actions:
        z = z + 0.6 * action
    return z

scores = []
for idx, action_seq in enumerate(candidates):
    z_terminal = rollout(z0.copy(), action_seq)
    score = float(np.linalg.norm(z_terminal - zg))
    scores.append((idx, z_terminal.round(3).tolist(), round(score, 3)))

print(scores)

The expected output is a ranked list of candidates with terminal latent states and distances. If two plans tie closely, that is not a bug, it is the planner telling you the current latent dynamics do not yet separate the futures strongly enough.

Why Small Robot Data Can Be Enough

The V-JEPA 2 paper's claim is not that 62 hours of robot data solve robotics by itself. The narrower claim is that a very large passive video prior can make the action-conditioning stage dramatically more sample-efficient. The robot data no longer has to teach the model all of visual world structure; it mainly has to teach how this embodiment's actions perturb that structure.

This division of labor is scientifically interesting because it is one of the clearest examples in the book where web-scale passive observation and small-scale interaction data play genuinely different roles.

Implementation Audit

Code Fragment 2 below defines the minimum audit record for an action-conditioned latent planning run.

# Save the contract for a V-JEPA 2 action-conditioned rollout test.
# A real experiment should keep the robot-data source and replanning
# horizon next to the final success metric.
from dataclasses import asdict, dataclass

@dataclass
class LatentPlanningAudit:
    backbone: str
    action_head: str
    robot_data_hours: float
    goal_type: str
    replanning_horizon: int
    metric: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

audit = LatentPlanningAudit(
    backbone="vjepa2_base",
    action_head="vjepa2_ac",
    robot_data_hours=62.0,
    goal_type="image_goal",
    replanning_horizon=8,
    metric="goal_reached_without_intervention",
)
print(audit.as_row())

The expected output is a compact audit dictionary. If your own experiment log cannot at least name these fields, the later success rate will be difficult to interpret.

The cleanest way to read V-JEPA 2 is as a two-stage decomposition: learn general predictive visual structure first, then learn how your action space moves that structure. For embodied AI, that is attractive because it separates abundant passive data from scarce interaction data while still keeping planning in the loop.