Section 41.4: Generating scenes and synthetic experience

Generate the whole future as one object, then bend it toward high reward with a guidance gradient; the plan and the optimizer become the same forward pass.

A Trajectory-Level Planner

Planning with generative models reframes the planner as a sampler. Diffuser (Janner et al., 2022) generates a full trajectory at once by diffusion, then steers that generation toward high reward with classifier guidance, so the act of planning and the act of optimizing collapse into the same denoising forward pass. Decision Diffuser instead conditions on a desired return-to-go for offline reinforcement learning, sampling trajectories that the dataset suggests will achieve that return. The same generative machinery also produces candidate futures and synthetic scenes that planners and learners can consume.

The catch is shared across all of these uses: generated trajectories and scenes help only when they stay near the real control regime, and the guidance that makes them high-reward can also pull them off the data manifold. The right framing is generation as a proposal engine whose proposals must survive a feasibility check, whether the proposal is a planned trajectory or a synthetic training scene.

Theory

Diffuser denoises an entire trajectory $\tau$ and biases sampling toward high reward using classifier (or classifier-free) guidance. At each reverse step the unconditional noise prediction is combined with a conditional one to form a guided estimate

$$ \tilde\epsilon_\theta = (1+w)\,\epsilon_\theta(\tau_t, t, c) - w\,\epsilon_\theta(\tau_t, t), $$

where $c$ is the conditioning (a goal or a high-return signal) and $w$ is the guidance weight. With $w=0$ the sample is unconditional; larger $w$ pushes harder toward the conditioned objective. The trade-off is direct: strong guidance shapes more goal-directed plans but can drag samples off the data manifold into dynamically infeasible trajectories, which is why a feasibility filter still sits downstream. Decision Diffuser uses this same conditioning mechanism with the return-to-go as $c$, turning offline reinforcement learning into conditional trajectory generation.

A practical cost note frames the rest of the chapter: generative planning pays 20 to 50 denoising steps at inference for every plan, against a single forward pass for a deterministic policy. That latency is the price of multimodality and guidance, and it is exactly what the speedup work in Section 41.5 attacks.

When generated trajectories are reused as training data, a dataset mixes real trajectories $\mathcal{D}_r$ with generated ones $\mathcal{D}_g$ under a weight $\alpha$, $\mathcal{L}(\theta)=\mathbb{E}_{\mathcal{D}_r}[\ell_\theta] + \alpha\,\mathbb{E}_{\mathcal{D}_g}[\ell_\theta]$. The hard part is not the mixture; it is ensuring $\mathcal{D}_g$ adds coverage near task-relevant but underrepresented states rather than injecting trajectories that violate real embodiment dynamics.

Worked Example

The probe below shows classifier-(free)-guidance trajectory planning and the guidance-weight trade-off Diffuser depends on. From the same noisy start it runs a guided denoising loop at three values of $w$, combining a conditional denoiser (pulls the endpoint to the goal) with an unconditional one (pulls toward an on-manifold prior). We report the endpoint, its distance to the goal, and whether it stays inside a feasible reach radius.

# Guided trajectory denoising (Diffuser-style).
# eps_tilde = (1 + w) eps_cond - w eps_uncond ; w trades goal-pull vs on-manifold pull.
import numpy as np

H = 5
goal = np.array([1.0, 0.0])
feasible_radius = 1.15           # endpoints beyond this are dynamically infeasible

def eps_uncond(tau):             # stand-in: pull toward a smooth on-manifold prior
    prior = np.stack([np.linspace(0.0, 0.6, H), np.zeros(H)], axis=1)
    return tau - prior

def eps_cond(tau, c):            # stand-in: pull the trajectory toward the goal c
    target = np.stack([np.linspace(0.0, c[0], H), np.linspace(0.0, c[1], H)], axis=1)
    return tau - target

for w in [0.0, 1.0, 4.0]:
    rng = np.random.default_rng(4)        # same start for every w
    tau = rng.normal(size=(H, 2)) * 0.3
    for _ in range(8):                     # guided reverse steps
        eps_tilde = (1 + w) * eps_cond(tau, goal) - w * eps_uncond(tau)
        tau = tau - 0.2 * eps_tilde
    endpoint = tau[-1]
    dist = float(np.linalg.norm(endpoint - goal))
    feasible = float(np.linalg.norm(endpoint)) <= feasible_radius
    print(f"w={w:>3}: endpoint={endpoint.round(3).tolist()} "
          f"dist_to_goal={dist:.3f} feasible={feasible}")

The three rows are the whole story of guidance. At $w=0$ the plan stays comfortably feasible but stops short of the goal; at $w=1$ it reaches the goal and is still feasible; at $w=4$ it overshoots far past the feasible reach radius. This is why Diffuser pairs guidance with a feasibility check: turning $w$ up makes plans more goal-directed right up to the point where they leave the manifold of trajectories the robot can actually execute. Decision Diffuser swaps the goal condition for a return-to-go target, but the same dial and the same risk apply.

Practical Recipe

1. Write the observation, action, horizon, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the maintained implementation only after the baseline behavior is understood.
4. Save one artifact containing configuration, seed panel, traces, metrics, and failure labels.
5. Run at least one perturbation test before trusting the result.

The most reliable pattern is to use generation as a data proposal engine and keep a conservative verifier downstream. Scene generators propose clutter, lighting, camera poses, or long-horizon futures; simulators, geometric filters, and held-out hardware tests decide whether those proposals are worth learning from.

For teaching and production alike, insist on one artifact that records real-sample count, synthetic-sample count, weighting, realism checks, and transfer outcome. Without that bookkeeping, synthetic data stories are nearly impossible to audit.

For Generating scenes and synthetic experience, keep one inspectable probe for the model assumption, then use maintained libraries without changing the artifact schema used for baseline comparison.

1. Write the observation, action, state estimate, success metric, and rejection criterion.
2. Run a deterministic smoke test on one seed and save the complete configuration.
3. Add one perturbation tied to the section topic: delay, noise, horizon length, contact change, distractor object, or generated-scene shift.
4. Compare only methods evaluated by the same script, split, seed panel, and metric definition.
5. Record a postmortem that assigns failures to perception, representation, dynamics, planning, control, data coverage, timing, or evaluation.

When Generating scenes and synthetic experience fails, do not collapse the result into a single method verdict. Assign the failure to the interface that broke, rerun one controlled perturbation, and keep the trace next to the metric. That habit turns a disappointing rollout into a reusable diagnostic asset.