Section 20.2: What transfers and what does not

A Careful Control Loop

For What transfers and what does not, sim-to-real transfer should name the randomized variables, simulator assumptions, real-world measurement, and demonstration-learning handoff in one transfer ledger.

An RL policy should be decomposed before deployment into representation, policy structure, value estimates, reward terms, low-level actuation, and evaluation metrics. These pieces do not deserve the same transfer claim because each one couples to different simulator assumptions.

The key question is practical: which learned quantities can be reused, which must be calibrated, and which must be thrown away before the first hardware trial?

Theory

Let a policy be decomposed as $\pi(a_t\mid z_t)$, where $z_t=f(o_{0:t})$ is the learned state representation. The representation $f$ may transfer if it captures geometry, contact phase, or goal relation. The action distribution $\pi$ may fail if the target robot has different torque limits, command latency, backlash, or compliance.

Reward models transfer even less automatically. A simulator reward can be dense, clean, and privileged, while the real robot only exposes sparse events, noisy force estimates, delayed vision, and safety interlocks. A reward term that was useful for training may be invalid as an evaluation metric.

Worked Example

Code Fragment 20.2.1 builds a small transfer ledger for a drawer-opening policy. The output distinguishes reusable task structure from components that need hardware calibration or safety constraints.

# Classify policy components before hardware deployment.
# The ledger prevents simulator-only conveniences from masquerading as transfer.
components = {
    "contact_phase_detector": "reuse",
    "camera_exposure_threshold": "recalibrate",
    "maximum_pull_force": "constrain",
    "privileged_hinge_angle_reward": "discard",
}

for name, decision in components.items():
    print(f"{name}: {decision}")

Expected output: each policy component has a deployment decision. A transfer plan that says "deploy the policy" without this inventory hides the most important engineering choices.

Practical Recipe

1. List the learned components: encoder, memory state, policy head, value head, reward terms, termination logic, and controller interface.
2. Classify each component as reuse, recalibrate, constrain, or discard.
3. Remove privileged simulator inputs from the deployed observation path.
4. Replace simulator rewards with hardware-measurable evaluation metrics.
5. Run a small hardware gate for every component marked reuse, especially if it touches force, contact, or delay.

The idea in this section becomes useful when each learned quantity has an explicit deployment status. A policy checkpoint is not a single artifact from a transfer perspective. It contains representations, action distributions, value estimates, normalization statistics, and assumptions about reward and termination.

The graduate-level habit is to separate four claims. The invariance claim says what remains true across sim and real. The calibration claim says what must be measured on hardware. The safety claim says what must be constrained before exploration. The evidence claim says which hardware gate proves the decision was justified.

A robust implementation starts with a transfer ledger and a hardware gate. The ledger records the deployment decision for each component. The gate records the smallest hardware test that can falsify that decision.

1. Create a component ledger before loading the trained policy on hardware.
2. Attach a validation gate to every reuse and recalibrate decision.
3. Prohibit privileged simulator channels from the hardware observation contract.
4. Save the ledger, gate result, trace, and failure label in one artifact.
5. Report transfer only for components whose evidence was collected under the same robot protocol.

When transfer fails, ask which ledger decision was wrong. A reuse failure means the task invariant was overestimated. A recalibration failure means the measurement protocol was too weak. A constrain failure means the safety envelope did not cover the robot's actual behavior. A discard failure means a simulator-only shortcut leaked into deployment.