Section 13.6: Randomization vs. realism; measuring transfer readiness

A Careful Control Loop

What This Section Builds

This section makes transfer readiness operational. It explains how to decide when to spend effort on broader randomization, better realism, or a hybrid that reconstructs measured factors and randomizes residual uncertainty.

The goal is a comparison that a skeptical reader can audit number by number: one task panel, one configuration, one seed policy, one metric definition, and one artifact containing all compared results.

Theory

A useful readiness score starts with construct matching. If the claim is about grasp success, compare grasp success on the same object panel, not detector average precision from one run and robot success from another. If the claim is about pose robustness, compare pose error under the same camera split, not a different render split.

Randomization is preferable when the real distribution is broad, uncertain, and too expensive to reconstruct precisely. Realism is preferable when measured details dominate failure, such as camera calibration, object scale, or contact geometry. A hybrid is often strongest: reconstruct what can be measured, then randomize residual uncertainty around it.

Worked Example

The following snippet computes a simple transfer-readiness comparison from one shared panel. The important detail is that randomization, realism, and hybrid scores are stored together rather than copied from separate experiments.

# Compare transfer strategies on one shared evaluation panel.
# Every number uses the same metric, scenes, seeds, and failure labels.
results = {
    "baseline": {"real_success": 0.58, "slip_failures": 21},
    "randomized": {"real_success": 0.71, "slip_failures": 12},
    "realistic": {"real_success": 0.67, "slip_failures": 15},
    "hybrid": {"real_success": 0.76, "slip_failures": 8},
}

for method, metrics in results.items():
    gain = metrics["real_success"] - results["baseline"]["real_success"]
    print(f"{method}: success={metrics['real_success']:.2f}, gain={gain:.2f}")

Practical Recipe

1. Choose the real or proxy panel that will define transfer readiness before training variants.
2. Use the same success metric, failure taxonomy, scene split, and seed policy for every method.
3. Compare baseline, randomization, realism, and hybrid strategies in one evaluation artifact.
4. Report real success, major failure labels, and the gap between simulator and real performance.
5. Treat any number from a different configuration as diagnostic context, not as part of the main comparison.

Randomization, realism, and hybrid strategies become useful when they are judged by the same closed-loop evidence. The artifact should include the simulator score, real score, gap, failure labels, and exact split identifiers.

The graduate-level habit is to separate three claims. The simulator claim says the method performs under synthetic conditions. The transfer claim says it performs on held-out real or proxy conditions. The readiness claim says the gap and failure labels are small enough for the next deployment step.

A robust implementation starts with a single comparison artifact. Code Fragment 2 records simulator score, real score, gap, metric, and split for one method, and the same schema should be used for every compared method.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

Expected output: the printed trace should expose method, simulator score, real score, metric, split, and sim-to-real gap. If one of those fields is missing, the example is not yet an evaluation artifact.

When a transfer-readiness comparison fails, first check whether the metrics are construct-matched and co-computed. Then inspect the failure labels: a large sim-to-real gap with many perception failures calls for rendering or sensor work, while a small gap with many contact failures points to dynamics, controller, or task coverage.