Section 7.7: Controllers vs. policies; when learning helps and when it makes control unsafe

A Careful Control Loop

This section develops the technical contract for Controllers vs. policies; when learning helps and when it makes control unsafe into a usable mental model. First we define the object of study, then we connect it to the agent loop, then we test it with a compact implementation.

The key question in Controllers vs. policies; when learning helps and when it makes control unsafe is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

For Controllers vs. policies; when learning helps and when it makes control unsafe, the practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

A classical controller is usually strongest when the goal, state, model class, and safety limits are explicit. A learned policy is strongest when perception, contact variation, human preference, or high-dimensional context is too complex to hand-code. The safest hybrid treats the learned policy as a proposal generator and the controller or safety filter as the executable contract.

Worked Example

The safest place for learning in a control loop is as a residual on top of a classical controller, wrapped by a safety filter. The command is \(u = u_\text{base}(x) + \pi_\theta(x)\): the base controller owns the nominal behavior, and the learned residual nudges it. A control barrier function (CBF) then enforces a hard safety set. For a barrier \(h(x)\ge 0\) (here, staying left of a wall), the filter requires \(\dot h(x) + \alpha h(x) \ge 0\) and projects the requested command onto the closest admissible one. Code Fragment 7.7.1 runs the same residual policy with and without the filter.

import numpy as np

# u = u_base(x) + pi_theta(x), guarded by a control barrier function.
# Plant: 1D mass. Barrier h(x) = x_wall - x >= 0  (stay left of the wall).
m, dt, x_wall, alpha = 1.0, 0.05, 1.0, 4.0

def u_base(x, v):   return 8.0 * (0.9 - x) - 4.0 * v   # nominal PD toward x=0.9
def pi_theta(x, v): return 5.0                          # learned residual (unsafe alone)

def cbf_filter(u, x, v):
    # h = x_wall - x, hdot = -v.  Enforce hddot + 2*alpha*hdot + alpha^2*h >= 0
    # with acceleration a = u/m affecting hddot = -a.
    h, hdot, a = (x_wall - x), -v, u / m
    margin = (-a) + 2 * alpha * hdot + alpha ** 2 * h
    if margin >= 0:
        return u, False
    u_safe = m * (2 * alpha * hdot + alpha ** 2 * h)    # minimal correction: margin -> 0
    return u_safe, True

for label, use_filter in [("no safety filter", False), ("with CBF filter", True)]:
    x = v = 0.0; interventions = 0; xmax = -9.0
    for _ in range(200):
        u = u_base(x, v) + pi_theta(x, v)
        if use_filter:
            u, did = cbf_filter(u, x, v); interventions += int(did)
        x += v * dt; v += (u / m) * dt
        xmax = max(xmax, x)
    breached = "BREACHED" if xmax > x_wall + 1e-3 else "safe"
    print(f"{label:>17}: max x={xmax:.3f} wall={x_wall} -> {breached}  interventions={interventions}")

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

Production Pattern

Controllers vs. policies; when learning helps and when it makes control unsafe sits inside the Part II robotics contract: geometry defines where things are, kinematics defines what motion is possible, dynamics defines what motion costs, control defines how errors are corrected, and sensing defines what the agent can know on time.

Compare controllers and learned policies only under the same sensors, action limits, disturbances, and safety filters. This makes the section useful to students, builders, and researchers at the same time: the idea has an intuitive role, a formal interface, a runnable check, and a failure mode that can be reproduced.

Use this recipe when turning Controllers vs. policies; when learning helps and when it makes control unsafe into code, a simulator experiment, or a robot diagnostic. The point is not to use every library. The point is to keep the hand-built baseline and the maintained-tool path comparable.

1. Write the control objective, measured state, actuator command, update rate, and saturation policy.
2. Run a step-response test before adding learning, with overshoot, settling time, and steady-state error logged.
3. Compare the hand controller with python-control, CasADi, Drake, do-mpc, or ROS 2 control on the same plant model.
4. Record latency, missed deadlines, saturation events, constraint violations, and recovery actions.
5. Only compare controllers and policies when they share sensors, action limits, disturbance tests, and safety checks.

A learned policy can hide an unsafe control interface until the disturbance changes. Check action limits, latency, recovery authority, safety-filter intervention rate, out-of-distribution observations, and fallback behavior before scaling training. For this section, first reproduce one controller-only case and one policy-proposal case under the same disturbance panel. If the two disagree, inspect whether learning improved perception or task selection, or merely bypassed a constraint that the classical controller was enforcing.