Section 6.5: Differentiable physics: what it buys you

A Careful Control Loop

This section develops the technical contract for Differentiable physics: what it buys you into a usable mental model. First we define the object of study, then we connect it to the agent loop, then we test it with a compact implementation.

The key question in Differentiable physics: what it buys you is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

For Differentiable physics: what it buys you, the practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Worked Example: Backprop Through a Rollout, Validated Against Finite Differences

A differentiable simulator exposes a discrete update $x_{k+1} = f_\Delta(x_k, u_k, \theta)$ and a scalar loss $L = \ell(x_T)$, then lets you compute sensitivities such as $\partial L / \partial u_k$ or $\partial L / \partial \theta$ by the chain rule through every step. The example optimizes the launch velocity of a projectile so it lands on a target, by differentiating the landing-error loss through a symplectic rollout. The non-negotiable discipline is to validate the analytic gradient against a centered finite difference before trusting it for optimization.

For a horizon of $T$ steps the reverse-mode gradient accumulates local Jacobians, $\tfrac{\partial L}{\partial x_0} = \big(\prod_{k} \tfrac{\partial x_{k+1}}{\partial x_k}\big)^\top \tfrac{\partial \ell}{\partial x_T}$. Here we differentiate by hand to keep the mechanism visible, then check it numerically.

import numpy as np

g = 9.81
dt, T = 0.01, 300            # 3 s rollout
target_x = 6.0               # land here on the ground (y = 0)

def rollout(v0):
    """Symplectic-Euler projectile from origin. Returns landing x and loss."""
    vx, vy = v0
    x, y = 0.0, 0.0
    for _ in range(T):
        vy = vy - g * dt     # gravity on vertical velocity
        x  = x + vx * dt
        y  = y + vy * dt
    loss = 0.5 * (x - target_x)**2
    return x, loss

def analytic_grad(v0):
    """d loss / d v0 by differentiating the closed-form rollout.
    With constant vx: x_T = vx * (T*dt). Loss = 0.5*(x_T - target)^2."""
    vx, vy = v0
    x_T = vx * (T * dt)
    err = x_T - target_x
    dloss_dvx = err * (T * dt)        # dx_T/dvx = T*dt
    dloss_dvy = 0.0                    # horizontal landing x independent of vy here
    return np.array([dloss_dvx, dloss_dvy])

def finite_diff_grad(v0, eps=1e-6):
    grad = np.zeros(2)
    for i in range(2):
        vp = v0.copy(); vp[i] += eps
        vm = v0.copy(); vm[i] -= eps
        grad[i] = (rollout(vp)[1] - rollout(vm)[1]) / (2 * eps)
    return grad

v0 = np.array([2.5, 1.0])
print("analytic grad     :", analytic_grad(v0))
print("finite-diff grad  :", finite_diff_grad(v0))
print("max abs diff      :", np.max(np.abs(analytic_grad(v0) - finite_diff_grad(v0))))

# Gradient descent on v0 to hit the target.
v = np.array([2.5, 1.0]); lr = 0.05
for step in range(200):
    v = v - lr * analytic_grad(v)
x_final, loss = rollout(v)
print(f"optimized vx={v[0]:.4f}  landing x={x_final:.4f}  loss={loss:.2e}")

The analytic and finite-difference gradients agree to within numerical tolerance, which is the green light to optimize: gradient descent then drives the landing point onto the target. This rollout is smooth, so the gradient is exact. The cautionary half of the lesson is that adding a contact event (a bounce, a grasp, a foot strike) makes $f_\Delta$ only piecewise smooth, and the same finite-difference check will then expose where the autodiff gradient is biased by contact smoothing or undefined across a mode boundary. Run the check on one smooth case and one contact case before optimizing a long horizon.

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

Production Pattern

Differentiable physics: what it buys you sits inside the Part II robotics contract: geometry defines where things are, kinematics defines what motion is possible, dynamics defines what motion costs, control defines how errors are corrected, and sensing defines what the agent can know on time.

Use differentiable physics when gradients answer a design question, and validate them against finite differences. This makes the section useful to students, builders, and researchers at the same time: the idea has an intuitive role, a formal interface, a runnable check, and a failure mode that can be reproduced.

Differentiable physics is useful when the question is not only "what happened?" but "which parameter, action, or design choice would make the outcome better?" Gradients can tune masses, friction coefficients, control sequences, morphology parameters, or policy inputs, but only if the gradient is a faithful derivative of the simulation the reader intends to trust.

Use this recipe when turning Differentiable physics: what it buys you into code, a simulator experiment, or a robot diagnostic. The point is not to use every library. The point is to keep the hand-built baseline and the maintained-tool path comparable.

1. Specify mass, inertia, actuator limits, contact model, timestep, and solver tolerance before running a rollout.
2. Run one free-motion test and one contact test with logged energy, constraint violation, and penetration depth.
3. Compare the hand calculation with MuJoCo, Drake, Pinocchio, or MJX on the same model and timestep.
4. Store solver settings, random seed, initial state, trajectory, and failure labels in one artifact.
5. Scale to Isaac Lab or GPU-parallel simulation only after a small model passes deterministic checks.

For Differentiable physics: what it buys you, distrust smooth simulation until the section-specific physical assumption has been stress-tested: timestep, contact stiffness, damping, friction, actuation, and energy behavior should each have a small diagnostic.