Section 20.4: Fine-tuning on hardware; safe real-world RL

A Careful Control Loop

For Fine-tuning on hardware; safe real-world RL, sim-to-real transfer should name the randomized variables, simulator assumptions, real-world measurement, and demonstration-learning handoff in one transfer ledger.

This section develops a hardware fine-tuning protocol. The protocol includes a frozen simulator-trained policy, a real-robot evaluation gate, a constrained update rule, a human or automated stop channel, and a rollback checkpoint. The point is to improve transfer without turning the robot into an uncontrolled training environment.

The key question is practical: what evidence lets the team safely move from evaluation-only rollouts to limited policy updates on hardware?

Theory

A hardware fine-tuning loop can be written as a constrained update: choose action $a_t \sim \pi_\phi(a_t\mid o_t)$ only if $g_i(o_t,a_t) \leq 0$ for every safety gate $i$. The gates may encode workspace limits, force limits, speed limits, joint-temperature limits, distance from humans, controller health, and intervention triggers.

The update rule should be smaller than the safety case. A common pattern is to freeze the perception stack and low-level stabilizer, then adapt a residual policy, gain schedule, or high-level action bias. That limits the blast radius of learning while still allowing hardware-specific improvement.

Worked Example

Code Fragment 20.4.1 implements a simple action gate for a residual pushing policy. The proposed residual is allowed only if speed, force, actuator delay, and workspace checks stay within the approved envelope.

# Gate a residual action before it reaches the real robot.
# A hardware fine-tuning loop updates only after safety checks pass.
proposal = {"speed_mps": 0.18, "force_n": 14.0, "delay_ms": 38, "workspace_ok": True}
limits = {"speed_mps": 0.20, "force_n": 15.0, "delay_ms": 45}

passes_gate = (
    proposal["workspace_ok"]
    and proposal["speed_mps"] <= limits["speed_mps"]
    and proposal["force_n"] <= limits["force_n"]
    and proposal["delay_ms"] <= limits["delay_ms"]
)

print(f"passes_safety_gate={passes_gate}")

Expected output: a hardware fine-tuning diagnostic reports whether the action passed the gate and which constraint would have blocked it. If blocked actions are not logged, the learner's data distribution cannot be audited.

Practical Recipe

1. Begin with evaluation-only hardware rollouts from the frozen sim-trained policy.
2. Define safety gates for workspace, speed, force, joint limits, temperature, actuator delay, human proximity, and controller health.
3. Choose the smallest trainable subset: residual action, gain schedule, adapter, or high-level command bias.
4. Set an intervention budget and rollback rule before the first update.
5. Report learning progress together with safety events, blocked actions, human interventions, and hardware resets.

The idea in this section becomes useful when the hardware protocol is explicit enough to audit. The protocol names the frozen policy checkpoint, the trainable residual, the gate conditions, the monitoring frequency, the intervention policy, and the rollback checkpoint. Without these details, hardware fine-tuning becomes an anecdote.

The graduate-level habit is to separate three claims. The improvement claim says the policy performs better. The safety claim says exploration stayed within the approved envelope. The deployment claim says the final policy remains robust under held-out initial conditions and delay tests.

A robust implementation starts with a safety gate document and a logging schema. The gate document states what can stop an action. The schema records what was proposed, what was blocked, what was executed, and whether the policy was updated afterward.

1. Run the frozen policy under the same gates used for fine-tuning.
2. Train only after the frozen-policy intervention rate is below the predeclared threshold.
3. Update the smallest residual or adapter that can plausibly fix the observed gap.
4. Checkpoint before every update batch and define automatic rollback triggers.
5. Evaluate the final policy on held-out starts, delay perturbations, and safety-envelope edges.

When hardware fine-tuning fails, assign the failure to one of four categories: unsafe proposal, wrong gate, harmful update, or misleading reward. Then rerun the frozen checkpoint against the same starts to decide whether the update caused the failure or merely exposed a preexisting transfer gap.