Section 13.5: real2sim2real and asset/scene reconstruction

A Careful Control Loop

What This Section Builds

This section makes real2sim2real operational. It distinguishes reconstruction inputs, such as scans, meshes, camera calibration, object dimensions, and contact measurements, from evaluation outputs such as real success, pose error, slip, collision, or recovery rate.

The goal is to build a simulator that is anchored to reality without becoming a benchmark mirror. A digital twin should explain what was measured, what was inferred, and what remains randomized.

Theory

The real2sim2real loop has four steps. First, measure the real environment: geometry, material appearance, camera calibration, object dimensions, mass, friction, and task layout. Second, reconstruct an asset or scene representation. Third, randomize residual uncertainty around the reconstruction, because the scan is never the whole world. Fourth, evaluate transfer on real conditions held out from reconstruction.

The critical split is calibration versus evaluation. Calibration data may tune mesh scale, camera intrinsics, contact parameters, and lighting priors. Evaluation data must remain separate, otherwise the digital twin can memorize the test room, test object, or test camera path.

Worked Example

The following snippet turns measured object dimensions into a residual randomization range. The simulator is anchored to the real measurement, but the training distribution still covers small reconstruction and manufacturing errors.

# Convert real calibration measurements into residual randomization ranges.
# The holdout object is excluded, so range fitting does not leak evaluation data.
calibration_lengths_cm = [9.8, 10.1, 10.0, 9.9]
margin_cm = 0.3

low = min(calibration_lengths_cm) - margin_cm
high = max(calibration_lengths_cm) + margin_cm
holdout_object = "object_E, unseen during reconstruction"

print(f"length_range_cm=({low:.1f}, {high:.1f})")
print(f"holdout={holdout_object}")

Practical Recipe

1. Split real data into calibration, validation-for-debugging, and final holdout sets before reconstruction.
2. Record which assets were scanned, which parameters were fitted, and which parameters remain randomized.
3. Randomize residual uncertainty around reconstructed values rather than treating the reconstruction as exact.
4. Evaluate on real objects, scenes, or camera paths absent from the reconstruction set.
5. Report failures as reconstruction error, residual range miss, simulator physics miss, perception miss, or policy miss.

real2sim2real and asset reconstruction become useful when the simulator is treated as an estimate with uncertainty. The reconstructed asset is the center of the distribution, not the full distribution.

The graduate-level habit is to separate three claims. The reconstruction claim says the digital twin matches measured calibration data. The residual claim says remaining uncertainty is randomized across plausible ranges. The evidence claim says final real performance is measured on conditions excluded from reconstruction.

A robust implementation starts with a provenance record. Code Fragment 2 records the calibration source, residual range, final holdout, and leakage guard beside the transfer metric.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

Expected output: the printed trace should expose calibration source, residual randomization, transfer metric, and leakage guard. If one of those fields is missing, the example is not yet an evaluation artifact.

When real2sim2real fails, separate scan error, mesh simplification error, contact-parameter miss, residual range miss, perception mismatch, and policy mismatch. Then adjust only the suspected reconstruction or residual range and rerun on the same final holdout. This preserves the evidence value of the holdout panel.