Section 17.2: Learning to walk in minutes: the parallel-RL recipe

A Careful Control Loop

For Learning to walk in minutes: the parallel-RL recipe, GPU RL depends on simulator fidelity, PPO rollout semantics, reward terms, and reset distribution being versioned in the same training artifact.

This section develops the parallel-RL recipe behind fast locomotion training. We focus on the control decisions that make a huge rollout useful: short horizons, many environments, stable normalization, randomized starts, and a held-out evaluation panel.

The key question is practical: how much simulated walking experience reaches the learner per update, and how do we keep that experience fresh enough that PPO is still optimizing the policy that collected it?

Theory

Suppose a locomotion run uses $N=4096$ environments and a rollout horizon of $T=24$ control steps at 50 Hz. One PPO update then contains $98{,}304$ transitions, but each environment contributes only $0.48$ seconds of fresh behavior before the policy updates.

This is the central tradeoff. Larger $N$ increases batch size without lengthening policy lag. Larger $T$ improves temporal credit assignment but lets the rollout drift farther from the policy that will be updated.

Worked Example

Code Fragment 17.2.1 calculates the sample geometry for a typical fast locomotion run. The aggregate robot-seconds are large, but the per-environment horizon stays short to control policy lag.

# Size a parallel locomotion PPO update from first principles.
# Short per-environment rollouts keep policy lag small while N supplies scale.
num_envs = 4096
horizon = 24
control_hz = 50
minibatches = 8
ppo_epochs = 5

samples = num_envs * horizon
seconds_per_env = horizon / control_hz
aggregate_robot_seconds = samples / control_hz
minibatch_size = samples // minibatches
sample_reuse = ppo_epochs

print(f"samples per update: {samples:,}")
print(f"seconds per env before update: {seconds_per_env:.2f}")
print(f"aggregate robot-seconds: {aggregate_robot_seconds:,.1f}")
print(f"minibatch size: {minibatch_size:,}")
print(f"sample reuse per rollout: {sample_reuse} PPO epochs")

Expected output: the trace should make policy lag visible. If a recipe reports only total samples and hides horizon, minibatch size, and PPO epochs, it is hard to tell whether the update is fresh or over-reused.

Practical Recipe

1. Start with a short horizon such as 16 to 32 control steps, then increase only if credit assignment clearly needs it.
2. Use enough environments to keep minibatches large without reusing stale data too many times.
3. Randomize terrain, commands, friction, mass, latency, and pushes by seed family, not by ad hoc global switches.
4. Track reward terms separately so a standing-still policy cannot hide behind a shaped reward total.
5. Evaluate without exploration noise on held-out seeds every fixed number of updates.

The recipe becomes useful when every speed claim is paired with a freshness claim. A wide rollout with 10 PPO epochs may train quickly, but it also asks the learner to reuse behavior from an older policy. A narrower rollout with fewer epochs may use fresher data but underfill the GPU.

The graduate-level habit is to report the whole recipe, not the headline time. A reproducible locomotion result names task randomization, reset curriculum, control frequency, horizon, action scaling, reward terms, policy architecture, normalization, evaluation seeds, and hardware.

A robust implementation starts by freezing the recipe fields that affect learning speed. The example below records the batch geometry and policy-lag budget next to the evaluation panel, so a later table can compare runs without mixing different configs.

1. Lock the control frequency, horizon, and action decimation before tuning reward.
2. Log every reward term and reset reason, not only the total return.
3. Keep normalization statistics versioned with the checkpoint.
4. Save evaluation videos or state traces on held-out seeds at fixed update intervals.
5. Compare recipes only when one evaluation script computes success, fall rate, and command error in one pass.

# Store the recipe fields that make a fast locomotion run auditable.
# Policy lag is controlled by horizon and sample reuse, not by env count alone.
from dataclasses import dataclass, asdict

@dataclass
class LocomotionRecipe:
    envs: int
    horizon: int
    minibatches: int
    ppo_epochs: int
    control_hz: int
    eval_seed_panel: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

recipe = LocomotionRecipe(
    envs=4096,
    horizon=24,
    minibatches=8,
    ppo_epochs=5,
    control_hz=50,
    eval_seed_panel="terrain_v3_holdout_0000_0255",
)
print(recipe.as_row())

When the policy learns quickly and transfers poorly, inspect the reward terms and reset reasons before changing the network. A policy may have learned to minimize falls by exploiting termination, to track commands only on easy terrain, or to depend on privileged simulator signals that will not exist at deployment.