Section 14.4: Model-free vs. model-based; on- vs. off-policy

A Careful Control Loop

This section links back to Chapter 7: Control for AI Practitioners and Chapter 10: Environments with Gymnasium and PettingZoo, then prepares the policy-gradient work in Chapter 15: Policy Gradient Methods and PPO. The section separates two axes that are often blurred: whether the learner uses an explicit dynamics model, and whether it learns from data generated by the same policy it is improving.

This section develops the technical contract for model-free vs. model-based learning and on-policy vs. off-policy data. First we define each axis, then we introduce discounted occupancy, then we compute how two policies can see different parts of the same environment.

The key question is practical: is the learner improving from its own fresh rollouts, from another policy's data, or from an explicit model of what the world will do next?

Theory

A model-free method estimates a policy, value function, or action-value function without explicitly learning $P(s'\mid s,a)$ for planning. A model-based method learns or uses a transition and reward model, then chooses actions by planning, imagination rollouts, or dynamic programming. In robot tasks, model-based methods can reduce physical samples, but learned models can compound small prediction errors across imagined steps.

An on-policy method updates a target policy using data generated by that same policy. An off-policy method learns about a target policy $\pi$ from data generated by a behavior policy $\mu$. Off-policy learning is attractive because embodied data are expensive, but the mismatch between $\mu$ and $\pi$ must be measured rather than waved away.

The discounted occupancy measure makes this mismatch concrete:

$$d_\gamma^\pi(s,a)=(1-\gamma)\sum_{t=0}^{\infty}\gamma^t\Pr_\pi(S_t=s,A_t=a).$$

This quantity says how often policy $\pi$ visits each state-action pair when near-term visits receive more weight. If a dataset has high occupancy for careful approaches but the target policy wants fast contact-rich grasps, the dataset may contain too little evidence for the target behavior.

Worked Example

Code Fragment 1 computes a finite-horizon approximation to discounted occupancy for two policies in a two-state robot task. The behavior policy is cautious; the target policy is more aggressive about contact.

# Estimate discounted occupancy for behavior and target policies.
# The mismatch shows where off-policy data may undersupport learning.
gamma = 0.9
states = ["approach", "contact"]
actions = ["slow", "fast"]
policies = {
    "behavior_mu": {"approach": {"slow": 0.8, "fast": 0.2}, "contact": {"slow": 0.7, "fast": 0.3}},
    "target_pi": {"approach": {"slow": 0.3, "fast": 0.7}, "contact": {"slow": 0.2, "fast": 0.8}},
}

def next_state(state, action):
    return "contact" if action == "fast" else state

for name, policy in policies.items():
    state_distribution = {"approach": 1.0, "contact": 0.0}
    occupancy = {(s, a): 0.0 for s in states for a in actions}
    for t in range(5):
        weight = (1 - gamma) * (gamma ** t)
        next_distribution = {"approach": 0.0, "contact": 0.0}
        for state, state_prob in state_distribution.items():
            for action, action_prob in policy[state].items():
                prob = state_prob * action_prob
                occupancy[(state, action)] += weight * prob
                next_distribution[next_state(state, action)] += prob
        state_distribution = next_distribution
    print(name, {f"{s}/{a}": round(v, 3) for (s, a), v in occupancy.items()})

The expected output shows the coverage mismatch directly in the occupancy mass. behavior_mu spends most of its discounted weight on cautious approach actions, while target_pi places much more mass on contact/fast, which is exactly where off-policy support becomes weakest.

The example also clarifies model-based learning. If a reliable model predicted the transition from `approach` to `contact`, the learner could plan contact behavior before trying every variant on hardware. If the model predicts contact poorly, planning amplifies the error.

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

The two axes in this section answer different failure questions. If a model-based controller fails, inspect transition prediction, reward prediction, planning horizon, and model rollout error. If an off-policy learner fails, inspect whether the replay buffer actually covers the target policy's state-action occupancy.

For embodied agents, the strongest design often mixes categories. A system may learn a model for short-horizon contact prediction, learn a model-free value function for policy improvement, and use off-policy data from demonstrations. The label matters less than the evidence that each data source supports the update it is asked to make.

A robust implementation logs data provenance. Every transition should record the behavior policy, policy version, simulator or hardware source, reward version, and any model-generated rollout flag. Without those fields, an off-policy experiment cannot prove which distribution produced the evidence.

1. State whether the update is on-policy or off-policy.
2. State whether planning uses a learned, analytic, or simulator model.
3. Estimate occupancy coverage for important state-action regions.
4. Evaluate model rollout error separately from policy return.
5. Compare algorithms only on one environment panel and one metric definition.

When a model-based method fails, run one-step prediction checks before judging the planner. When an off-policy method fails, inspect the occupancy mismatch before tuning the loss. These two diagnostics isolate different root causes.