Section 20.3: Domain randomization, system identification, adaptation (RMA)

A Careful Control Loop

For Domain randomization, system identification, adaptation (RMA), sim-to-real transfer should name the randomized variables, simulator assumptions, real-world measurement, and demonstration-learning handoff in one transfer ledger.

This section develops the technical contract for dynamics randomization. The parameters of interest include mass, inertia, center of mass, joint damping, motor strength, actuator delay, contact friction, restitution, sensor noise, and terrain geometry. Each parameter should have a reason for its range.

The key question is practical: should we make the training distribution wider, identify the real parameter more carefully, or build an adaptation module that tracks the parameter during rollout?

Theory

Let $\theta$ denote simulator dynamics parameters such as mass, friction, damping, and delay. Domain randomization trains a policy over $\theta \sim p_{\text{train}}(\theta)$ rather than one fixed simulator. System identification estimates $\hat{\theta}$ from hardware traces. Residual randomization then trains or evaluates over $\theta \sim p(\theta\mid \hat{\theta}, \Sigma)$, where $\Sigma$ represents what the measurement still cannot pin down.

Rapid motor adaptation adds an online inference loop. A base policy receives observations and an adaptation vector $z_t$ inferred from recent state-action history. If the floor becomes slippery or the payload changes, $z_t$ should move before the robot falls, giving the same base policy a different dynamics context.

Worked Example

Code Fragment 20.3.1 computes residual randomization ranges after a simple identification pass. The measured robot centers the range, while uncertainty keeps the policy from becoming brittle.

# Center randomization around identified hardware parameters.
# Residual width records uncertainty instead of pretending calibration is exact.
identified = {"friction": 0.62, "motor_scale": 0.91, "delay_ms": 34}
residual_width = {"friction": 0.08, "motor_scale": 0.05, "delay_ms": 8}

for parameter, center in identified.items():
    width = residual_width[parameter]
    low = center - width
    high = center + width
    print(f"{parameter}: sample from [{low:.2f}, {high:.2f}]")

Expected output: each randomized dynamics parameter has a center and a residual width. If the width is not justified by measurement noise, hardware variation, or unmodeled effects, the randomization range is a guess rather than an experimental design choice.

Practical Recipe

1. Start with a parameter manifest: mass, inertia, friction, damping, motor strength, actuator delay, sensor noise, and terrain properties.
2. Mark each parameter as measured, estimated, randomized, adapted online, or held fixed.
3. Use system identification to center parameters that can be measured from hardware traces.
4. Apply residual randomization only to the uncertainty left after identification.
5. Evaluate policies on held-out dynamics combinations, then run a small real-robot gate before scaling hardware trials.

The idea in this section becomes useful when it is tied to a parameter contract. The contract names each dynamics parameter, its unit, its source, its training distribution, its hardware estimate, and the reason its range is safe. Without that contract, a robust policy can be impossible to reproduce or diagnose.

The graduate-level habit is to separate three claims. The robustness claim says training variation helps the policy tolerate dynamics change. The identification claim says hardware traces support a parameter estimate. The adaptation claim says recent rollout history contains enough information to infer the remaining latent dynamics.

A robust implementation starts with the parameter manifest, not the training command. The manifest states which parameters are randomized, which are identified from hardware, which have residual uncertainty, and which are inferred online by the adaptation module.

1. Write parameter ranges with units and evidence source.
2. Run system identification for measurable quantities before widening ranges by hand.
3. Train with held-out dynamics combinations that were not used for policy updates.
4. Evaluate residual randomization separately from broad randomization.
5. For RMA-style policies, log the adaptation vector and correlate it with measured terrain, payload, or actuator changes.

When a randomized policy fails, first check whether the failing parameter was outside the training range, inside the range but underrepresented, measured incorrectly, or hidden from the adaptation module. Each answer leads to a different repair: widen, resample, identify, or change the adaptation input.