Section 37.4: Imagination rollouts

Imagination helps only while the imagined data still resembles a world the policy will actually visit.

A Budget-Conscious MPC Loop

Short Rollouts, Big Consequences

In MBPO-style learning, real states from the replay buffer seed short model-generated rollouts. Those imagined transitions augment policy learning while limiting compounding error. The core trade-off is simple: more imagined data can accelerate learning, but only if rollout length stays inside the model's trusted region.

One useful mental model is

$$ \mathcal{D}_{\text{train}} = \mathcal{D}_{\text{real}} \cup \mathcal{D}_{\text{model}}^{(h)}, $$

where the imagination horizon $h$ is deliberately small. This keeps model-generated states near the support of real experience.

That support argument is the mechanism, not a stylistic preference. The learner is allowed to recycle real states into nearby imagined futures because the model has seen enough neighboring transitions to stay locally coherent. Once synthetic states start seeding further synthetic states, the training set drifts toward parts of state space that were never grounded by real interaction, and value estimates can become systematically optimistic.

Worked Probe

The probe below logs how many synthetic transitions are produced from a replay batch under different imagination horizons. It shows why horizon choice changes dataset composition so quickly.

# Count imagined transitions produced from one replay batch.
replay_batch = 128
horizons = [1, 3, 5]
imagined = {h: replay_batch * h for h in horizons}
ratio_to_real = {h: round(imagined[h] / replay_batch, 1) for h in horizons}
print({"imagined_transitions": imagined, "ratio_to_real": ratio_to_real})

Common Failure Modes

A short-horizon imagination pipeline can still go wrong in three ways. The model may be locally biased around precisely the states that matter for reward improvement. The policy may overfit to synthetic states that look easy under the model but are rarely visited in reality. Or the training loop may silently let synthetic transitions dominate the replay mixture. All three failures create the same surface symptom, a policy that looks data efficient but degrades sharply under real rollouts.

The fix is not to abandon imagination, but to instrument it. Save the source state for each imagined rollout, the horizon used, the ratio of synthetic to real updates, and at least one replayed failure trajectory where the imagined branch misled the learner. That turns a vague trust problem into a concrete evidence trail.