Section 1.4: Physical vs. simulated embodiment

"Simulation gives you replay. Hardware tells you which assumptions survived contact."

Section 1.4

Two worlds, one policy

Treat the simulator and the robot as two transition processes that share an action space. The simulator advances state by a declared model $T_{\text{sim}}(s' \mid s, a)$: equations the modeler wrote down for rigid-body dynamics, a contact solver, a friction law, and a sensor model. The physical robot advances by $T_{\text{real}}(s' \mid s, a)$, which no one ever wrote down in full. The same policy $\pi$ induces a trajectory distribution in each, and therefore two values of the task functional $J(\pi)$ from Section 1.1. The reality gap is their difference,

$$\Delta(\pi) = J_{\text{sim}}(\pi) - J_{\text{real}}(\pi).$$

This is the central object of the section. It is positive when a policy looks better in simulation than on hardware, which is the usual case, and it is measured in the units of $J$ (success rate, return, completion time), not asserted as a caveat. A sim-to-real method is good exactly to the extent that it makes $\Delta(\pi)$ small for the policies you actually deploy, and the honest way to report transfer is to give $J_{\text{sim}}$, $J_{\text{real}}$, and $\Delta$ together, computed on the same policy and the same task definition.

What simulation buys, and at what fidelity

Simulation is attractive for reasons that are concrete and quantifiable. It runs faster than wall-clock time, often by orders of magnitude, and it runs thousands of environments in parallel on a GPU, so a learning run that would take physical years finishes in hours. It offers perfect resets: every episode can start from an identical state, which makes counterfactual experiments (change one variable, hold the rest) possible in a way hardware never allows. It exposes ground-truth state (exact poses, contact forces, object masses) that on a robot must be estimated and is partly unobservable. And its failures are free: a simulated robot can fall ten thousand times with no broken gearbox and no safety incident.

What it does not give for free is agreement with the physical world. The divergence has named sources, and they map onto three fidelity dimensions worth keeping separate:

• Physical fidelity covers dynamics: contact and friction are stiff, discontinuous, and the hardest thing for any solver to get right; restitution, compliance, joint backlash, motor torque saturation, and actuation latency are routinely simplified or omitted.
• Visual fidelity covers the appearance gap that a vision policy sees: lighting, materials, textures, reflections, and the difference between a rendered image and a camera frame with real noise, motion blur, and rolling shutter.
• Behavioral fidelity covers whether the agent makes the same decisions and the same mistakes in both worlds. This is the only dimension that ultimately matters, and the one that physical and visual fidelity are merely means toward. A scene can look photorealistic and still produce the wrong failure.

These trade against speed. A high-fidelity contact solver with fine time steps and a path-traced renderer is slower per step, which shrinks the parallelism that made simulation attractive in the first place. The speed-fidelity tradeoff is the practitioner's core dial: you buy fidelity with compute, and past some point the marginal fidelity does not change which policy you would deploy, so paying for it is waste.

Closing the gap: identification and randomization

Two strategies dominate, and they pull in opposite directions. System identification narrows the simulator onto the specific robot: measure masses, friction coefficients, motor constants, and latencies, then fit $T_{\text{sim}}$ to match logged hardware rollouts. It shrinks $\Delta$ by making the model true. Its limit is that you can only identify what you thought to measure, and the residual unmodeled physics is exactly what bites on deployment. Domain randomization takes the opposite bet: rather than match one robot, sample the uncertain parameters (friction, mass, latency, lighting, textures) over a wide range during training, so the policy must be robust to all of them and treats the real world as just one more sample from the training distribution. It trades peak in-distribution performance for transfer that does not require an accurate model. In practice they combine: identify what you can measure, randomize over what you cannot. Chapter 13 develops simulation and the physics engines; Chapters 20 and 43 develop domain randomization and sim-to-real transfer in depth.

A reality gap you can run

The code below is the smallest honest demonstration of $\Delta$. A 1D braking controller must stop a sliding mass before a wall. It plans its braking distance from an assumed friction coefficient (the value it was tuned on, $\mu = 0.8$) and brakes accordingly. We then evaluate the identical controller, with no retuning, on hardware-like surfaces where the true friction is lower. The controller is correct in its own world and crashes in the other, and we read off the reality gap as the drop in success rate.

# Reality gap for a 1D braking controller tuned to one friction coefficient.
# Same policy, two worlds: it succeeds where mu matches and overshoots where mu drops.
import numpy as np

def overshoot(mu_true, mu_assumed, v0=4.0, dt=0.005, g=9.81, margin=0.30):
    # The controller knows the wall position and plans its braking distance from
    # its ASSUMED friction. It must stop on or before the wall (overshoot <= 0).
    planned_stop = v0**2 / (2.0 * mu_assumed * g)
    wall = planned_stop + margin
    brake_point = wall - planned_stop - v0 * dt   # one-step lead removes integration bias
    x, v = 0.0, v0
    while v > 1e-4:
        decel = mu_true * g if x >= brake_point else 0.0
        v = max(0.0, v - decel * dt)
        x += v * dt
    return x - wall                               # > 0 means it crossed the wall (a crash)

mu_nominal = 0.8                                   # the value the controller was tuned on
print(f"{'true mu':>8} {'overshoot (m)':>14} {'outcome':>8}")
for mu_true in (0.8, 0.6, 0.4, 0.3):
    ov = overshoot(mu_true, mu_assumed=mu_nominal)
    print(f"{mu_true:>8.2f} {ov:>14.3f} {'ok' if ov <= 0 else 'CRASH':>8}")

# Reality gap as a scalar: success at the tuned value vs. over a realistic friction band.
succeeds = lambda mu: overshoot(mu, mu_nominal) <= 0
J_sim  = float(np.mean([succeeds(0.8) for _ in range(50)]))                 # eval where it was tuned
J_real = float(np.mean([succeeds(mu) for mu in np.linspace(0.3, 0.8, 50)])) # eval over the real band
print(f"\nJ_sim  (mu fixed at 0.80)  = {J_sim:.2f}")
print(f"J_real (mu in [0.30,0.80]) = {J_real:.2f}")
print(f"reality gap  Delta         = {J_sim - J_real:.2f}")

Which fidelity dimension transfers what

The table separates the three fidelity dimensions and states, for each, what a simulator typically gets right, where the gap usually opens, and which gap-closing strategy applies. It is a planning aid: locate the claim you want to make in the left column and read across.