Section 16.5: Sample efficiency and off-policy failure modes

A Careful Control Loop

For Sample efficiency and off-policy failure modes, off-policy learning depends on replay semantics, environment API, target computation, and GPU-scale batching being fixed before comparison with policy-gradient methods.

Sample efficiency is the promise of off-policy RL: learn more from each transition by reusing data. The danger is that reused data may come from a different policy, a different simulator condition, or a different phase of the robot's learning history.

This section develops the failure map. We distinguish useful reuse from distribution mismatch, show why off-policy correction can become high variance, and name the embodied cases where bootstrapping turns missing coverage into confident value errors.

Theory

Off-policy learning trains a target policy using data collected by a behavior policy. If the behavior policy $\mu$ and target policy $\pi$ differ, an importance ratio can correct expectations in the simplest setting:

$$\rho_t = \frac{\pi(a_t|s_t)}{\mu(a_t|s_t)}$$

Large ratios mean the target policy strongly prefers an action that the behavior policy rarely took. That can reduce bias, but it increases variance. In long-horizon embodied tasks, multiplying many ratios can make estimates unusably noisy, so practical systems clip, truncate, or avoid explicit correction by using value-based bootstrapping.

Bootstrapping has its own failure mode. The critic can assign high value to state-action pairs that are out of distribution, because no transition in replay contradicts the estimate. This is extrapolation error: the model sounds certain exactly where the data is thin.

Worked Example

Code Fragment 1 computes importance ratios for three logged actions. The third transition is dangerous because the target policy assigns high probability to an action that the behavior policy rarely selected.

# Compute off-policy correction ratios for logged actions.
# Large ratios identify target-policy decisions with weak behavior-policy support.
logged = [
    {"action": "slow_push", "pi": 0.40, "mu": 0.50},
    {"action": "lift", "pi": 0.20, "mu": 0.25},
    {"action": "fast_recovery", "pi": 0.30, "mu": 0.03},
]

for row in logged:
    ratio = row["pi"] / row["mu"]
    clipped = min(ratio, 2.0)
    print(row["action"], f"rho={ratio:.1f}", f"clipped={clipped:.1f}")

The expected output shows mild correction for the first two logged actions and an extreme mismatch for fast_recovery. A ratio of 10.0 means the target policy wants that action far more often than the behavior policy ever demonstrated it, so clipping to 2.0 is a variance-control patch, not proof of support.

In an embodied dataset, that weak evidence is not an abstract statistical problem. It may mean the robot almost never attempted the emergency recovery motion during collection, yet the learned policy now depends on it during deployment.

Practical Recipe

1. Define sample efficiency as return or success per environment interaction, not per gradient step.
2. Tag replay by behavior policy, task condition, and collection time.
3. Measure policy-data mismatch with action coverage, importance ratios, or nearest-neighbor support.
4. Track critic uncertainty or critic disagreement on target-policy actions.
5. Report failure modes by distribution shift: observation, action, dynamics, reward, or termination.

Sample efficiency has to be construct-matched. If method A uses 10,000 real robot steps and method B uses 10,000 simulator steps plus a million logged transitions, the comparison is not a single sample-efficiency number. It is a data-regime comparison that must report every source of experience.

Off-policy failure analysis starts by asking which distribution changed. Observation shift changes what the encoder sees. Action shift changes which controls the critic must evaluate. Dynamics shift changes the consequence of the same action. Reward shift changes which behavior the value function calls good. Termination shift changes which failures are hidden by early episode endings.

A robust off-policy implementation records data provenance, not only return. Code Fragment 2 builds one artifact row that ties a sample-efficiency claim to the exact interaction and replay counts behind it.

1. Count real environment steps, simulator steps, replayed transitions, and gradient updates separately.
2. Store behavior-policy tags for every replay source.
3. Report action-support warnings for target-policy actions with weak coverage.
4. Evaluate the current policy on a fixed perturbation panel, not on replay alone.
5. Keep negative diagnostics in the registry even when only significant wins enter the paper tables.

# Build one sample-efficiency audit record.
# Separating data sources prevents invalid apples-to-oranges comparisons.
from dataclasses import dataclass, asdict

@dataclass
class SampleEfficiencyAudit:
    algorithm: str
    real_steps: int
    sim_steps: int
    replay_samples: int
    gradient_updates: int
    weak_action_support: str
    evaluation_panel: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = SampleEfficiencyAudit(
    algorithm="TD3",
    real_steps=0,
    sim_steps=50000,
    replay_samples=1000000,
    gradient_updates=200000,
    weak_action_support="fast_recovery torque range",
    evaluation_panel="mass_friction_delay_v1",
)
print(record.as_row())

The expected output should be read as an accounting record, not a performance claim. It says this result used only simulator interaction, extremely heavy replay reuse, and still has a named weak-support region, so any sample-efficiency comparison must keep those data sources and support warnings attached to the score.

When an off-policy method fails, do not start by blaming the algorithm name. First identify whether the bad value came from behavior-policy mismatch, missing action support, stale dynamics, reward mislabeling, termination bias, or critic extrapolation. Then rerun one matched perturbation panel with coverage diagnostics enabled.