Section 41.2: Diffuser and Decision Diffuser

Do not predict the next action; predict the next chunk of actions, then denoise the whole chunk until it is consistent with what the camera just saw.

An Action-Diffusion Policy

Diffuser and Decision Diffuser solve related but different planning problems. Diffuser samples trajectories that satisfy start-state and goal constraints, while Decision Diffuser samples trajectories that are also conditioned on desired return or value targets. Both belong to a broader family of action-diffusion methods whose practical robotics workhorse is Diffusion Policy: instead of denoising a full offline trajectory, it denoises a short chunk of future actions conditioned on the latest visual observation, then executes part of that chunk before replanning.

For robotics, the difference between these methods matters whenever the task objective is richer than "reach the goal." Return conditioning can favor high-reward but risky plans unless it is paired with explicit feasibility or safety filtering, and observation-conditioned action chunking trades a longer planning horizon for tight, reactive control at execution time.

Theory

Action-diffusion methods are trained with the denoising score-matching loss. Given a clean action sequence $\tau_0$ conditioned on observation $o$, sample a timestep $t$ and Gaussian noise $\epsilon$, form the noised sample using the forward closed form, and ask the network to predict the noise it sees:

$$ \mathcal{L} = \mathbb{E}_{t,\tau_0,\epsilon}\left[\left\lVert \epsilon - \epsilon_\theta\!\left(\sqrt{\bar\alpha_t}\,\tau_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\ t,\ o\right)\right\rVert^2\right]. $$

This is a regression on noise, not on actions, which is what lets the model represent a multimodal action distribution: many distinct $\tau_0$ values are consistent with the same observation, and the noise objective does not collapse them to an average. Diffuser uses the same loss over full trajectories $\tau=(s_0,a_0,\dots,s_H,a_H)$ conditioned on state or goal; Decision Diffuser adds a return target $\hat R$ to the conditioning $c$, turning "sample a plausible future" into "sample a plausible future likely to achieve this return."

Sampling: DDPM vs DDIM. The trained denoiser can be sampled two ways. DDPM runs the full stochastic reverse chain, typically hundreds to a thousand steps, and injects fresh noise at each step. DDIM reinterprets the same model as a deterministic (non-Markovian) trajectory from noise to data, letting you skip steps and sample in roughly 10 to 50 iterations with little quality loss. DDPM gives more sample diversity; DDIM gives the speed a control loop needs.

Action chunking. Rather than denoising one action, Diffusion Policy predicts a horizon of $H$ future actions in a single denoising pass, executes the first $k < H$ of them open-loop, then re-observes and replans. Predicting a chunk gives temporal consistency (no jitter between consecutive single-step predictions); executing only $k$ and replanning every $k$ keeps the policy reactive to disturbances. The choice of $H$ and $k$ is the main knob trading smoothness against responsiveness.

Worked Example

The probe below shows how to structure a Diffusion Policy inference loop. It denoises an action chunk of shape $(H, A)$ from Gaussian noise over $T$ DDIM steps, conditioning every step on a fixed observation embedding, then executes the first $k$ actions. The denoiser here is a stand-in; in a real system it is a trained U-Net or transformer. The control logic around it, start from noise, iterate the reverse step, condition on the observation, slice the first $k$, is exactly what production code does.

# Diffusion Policy inference: denoise an action chunk conditioned on an observation.
# Loop structure is the point; eps_theta stands in for a trained noise predictor.
import numpy as np

H, A = 8, 2                              # predict H future actions of dim A
T = 10                                   # DDIM denoising steps
k = 2                                    # execute the first k, then replan
abar = np.linspace(0.95, 0.02, T)[::-1]  # signal retention, clean -> noisy
obs = np.array([0.6, -0.2])             # observation embedding (stand-in)
rng = np.random.default_rng(1)

def eps_theta(a_t, t, o):
    # Trained model would go here. Stand-in: actions track the observation.
    target = np.tile(o, (H, 1))
    return (a_t - np.sqrt(abar[t]) * target) / np.sqrt(1.0 - abar[t] + 1e-8)

a_t = rng.normal(size=(H, A))           # start from pure Gaussian noise
for t in reversed(range(T)):            # reverse chain: t = T-1 ... 0
    eps = eps_theta(a_t, t, obs)        # predict the noise, conditioned on obs
    a0_hat = (a_t - np.sqrt(1.0 - abar[t]) * eps) / np.sqrt(abar[t])
    if t > 0:                           # DDIM deterministic step toward a_{t-1}
        a_t = np.sqrt(abar[t-1]) * a0_hat + np.sqrt(1.0 - abar[t-1]) * eps
    else:
        a_t = a0_hat                    # final step lands in action space

action_chunk = a_t
print("predicted chunk (first 3 of H):", action_chunk[:3].round(3).tolist())
print("execute first k =", k, "actions:", action_chunk[:k].round(3).tolist())

The chunk converges to the observation-consistent target because the stand-in denoiser drives it there; a trained $\epsilon_\theta$ would instead produce a multimodal, observation-appropriate action sequence. The structural takeaways carry over unchanged: the observation is embedded once and reused across all $T$ steps, DDIM lets $T$ be small enough for control rates, and only $k$ of the $H$ predicted actions are executed before the loop re-observes and runs again.

Practical Recipe

1. Write the observation, action, horizon, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the maintained implementation only after the baseline behavior is understood.
4. Save one artifact containing configuration, seed panel, traces, metrics, and failure labels.
5. Run at least one perturbation test before trusting the result.

Diffuser is the cleaner choice when you want a multimodal proposal distribution over trajectories and you already have an external scorer or feasibility filter. Decision Diffuser is stronger when the return target is meaningful and stable enough to guide sampling, but it becomes fragile when reward shaping or offline support is poor.

For robotics, the best practice is to treat both as candidate generators, not final arbiters. Let them propose futures, then let explicit collision checks, dynamics checks, and latency budgets decide what can really be executed.

For Diffuser and Decision Diffuser, keep one inspectable probe for the model assumption, then use maintained libraries without changing the artifact schema used for baseline comparison.

1. Write the observation, action, state estimate, success metric, and rejection criterion.
2. Run a deterministic smoke test on one seed and save the complete configuration.
3. Add one perturbation tied to the section topic: delay, noise, horizon length, contact change, distractor object, or generated-scene shift.
4. Compare only methods evaluated by the same script, split, seed panel, and metric definition.
5. Record a postmortem that assigns failures to perception, representation, dynamics, planning, control, data coverage, timing, or evaluation.

When Diffuser and Decision Diffuser fails, do not collapse the result into a single method verdict. Assign the failure to the interface that broke, rerun one controlled perturbation, and keep the trace next to the metric. That habit turns a disappointing rollout into a reusable diagnostic asset.