Section 20.5: Measuring transfer performance

A Careful Control Loop

For Measuring transfer performance, sim-to-real transfer should name the randomized variables, simulator assumptions, real-world measurement, and demonstration-learning handoff in one transfer ledger.

This section develops a transfer evaluation contract. The contract includes task success, return, completion time, safety violations, intervention count, blocked-action count, reset count, hardware health, and failure category. Reporting only success hides how the success was purchased.

The key question is practical: do the metrics compare the same construct under the same protocol, or are we comparing a clean simulator score with a noisy hardware anecdote?

Theory

A basic transfer ratio is $\rho = S_{\text{real}} / S_{\text{sim}}$, where $S$ is the same success metric measured on matched task instances. If a policy succeeds in 90 percent of simulated trials and 63 percent of hardware trials, $\rho = 0.70$. The ratio is useful only when both rates use the same task family and success rule.

Good reports also include gap diagnostics: success gap $S_{\text{sim}} - S_{\text{real}}$, intervention rate, safety-violation rate, blocked-action rate, and failure-category distribution. These metrics tell the reader whether the policy failed because it could not solve the task, because it needed supervision, or because safety gates stopped unsafe commands.

Worked Example

Code Fragment 20.5.1 computes a small transfer report from matched simulated and real trials. Notice that success, interventions, and failure labels are computed from the same rows.

# Compute transfer metrics from one matched evaluation artifact.
# Success and safety numbers stay tied to the same task panel.
sim_success = [1, 1, 1, 1, 0]
real_success = [1, 0, 1, 1, 0]
interventions = [0, 1, 0, 0, 1]

sim_rate = sum(sim_success) / len(sim_success)
real_rate = sum(real_success) / len(real_success)
transfer_ratio = real_rate / sim_rate
intervention_rate = sum(interventions) / len(interventions)

print(f"sim_success={sim_rate:.2f}")
print(f"real_success={real_rate:.2f}")
print(f"transfer_ratio={transfer_ratio:.2f}")
print(f"intervention_rate={intervention_rate:.2f}")

Expected output: a transfer report should include the simulator rate, hardware rate, transfer ratio, and safety denominator. If the policy required interventions, that fact belongs next to the success rate, not in a separate paragraph.

Practical Recipe

1. Freeze the policy checkpoint and evaluation code before running the comparison.
2. Define the task panel, success rule, timeout, safety gates, and intervention policy in writing.
3. Compute simulator and hardware metrics from one artifact schema.
4. Report uncertainty with trial counts, paired panels, and confidence intervals or bootstrap intervals when sample size permits.
5. Include failure labels, videos or state traces, and all safety denominators in the same result package.

The idea in this section becomes useful when the result table is built from a single evaluation artifact. The artifact names the checkpoint, simulator version, robot configuration, task panel, safety gate, success rule, raw trials, and failure labels. Without that artifact, a transfer table can be impossible to reproduce or interpret.

The graduate-level habit is to separate three claims. The performance claim says the policy solved the task. The transfer claim says the same construct was evaluated in sim and real. The robustness claim says the result survives held-out dynamics, actuator delay, sensor noise, and initial-condition shifts.

A robust implementation starts with the result schema, not with a plotting script. The schema should force every row to contain the policy identifier, task instance, simulator or robot source, success label, safety events, failure label, and trace pointer.

1. Define a row schema before collecting trials.
2. Run simulator and hardware evaluation through adapters that emit the same schema.
3. Compute all metrics, including safety denominators, from that one table.
4. Attach trace pointers so failures can be replayed and relabeled.
5. Publish aggregate metrics only with trial counts, uncertainty, and the failure-label distribution.

When a transfer result disappoints, do not stop at the aggregate gap. Split failures by perception miss, state-estimation drift, contact slip, actuator delay, controller saturation, safety gate, timeout, and evaluator disagreement. The failure distribution is often more useful than the mean score.