Section 10.3: Reward design and termination

A Careful Control Loop

What This Section Builds

Reward design and episode endings become operational when the learning signal and the stopping reason are both explicit. A reward tells the learner what behavior is being reinforced, while terminated and truncated tell the learner why the episode stopped.

The goal is to stop treating all endings as equal. Reaching the goal, dropping the object, violating a safety boundary, and hitting a time limit require different labels in both the learning loop and the experiment report.

Theory

A reward should be tied to the task construct, not to whatever sensor is easiest to measure. In a reach task, distance-to-target may help shape learning, but the success condition should still state the task boundary: target reached within tolerance, object stable, safety constraints respected.

Gymnasium separates two ending flags because they answer different questions. terminated means the task's own terminal state was reached. truncated means an outside condition, usually a time limit, stopped the episode before the task itself ended.

Worked Example

Code Fragment 10.3.1 forces a time-limit truncation in CartPole-v1. The pole has not necessarily reached a terminal failure state, but the wrapper stops the episode because the external step budget is exhausted.

# Show that a time limit produces truncation, not task termination.
# Learning code should log which flag caused the episode boundary.
import gymnasium as gym

env = gym.make("CartPole-v1", max_episode_steps=3)
observation, info = env.reset(seed=5)
env.action_space.seed(5)

for step_index in range(5):
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)
    print(step_index + 1, terminated, truncated)
    if terminated or truncated:
        break

env.close()

The expected output keeps both ending flags false for the first two steps, then flips only truncated to true at step 3. That pattern means the episode stopped because the time budget expired, not because the task dynamics reached a terminal success or failure state.

Practical Recipe

1. Write the terminal success and terminal failure conditions before writing reward shaping.
2. Use terminated for task-defined endings such as success, irreversible failure, or safety violation.
3. Use truncated for external cutoffs such as time limits, evaluation budgets, or watchdog stops.
4. Log reward components in info during debugging, even if the trainer only consumes the scalar reward.
5. Report success rate, truncation rate, and failure category counts together.

Reward design is where a simulator becomes a teacher. If the scalar reward rewards the wrong proxy, the policy will optimize that proxy with more patience than the author expects. The environment should therefore store the reward decomposition and ending cause with each episode.

The graduate-level habit is to treat reward as a measurement model. The scalar is only a compressed signal. The full evidence artifact should retain the terms that explain the compression, especially when comparing methods across seeds or perturbations.

A robust implementation keeps reward computation auditable. Even when the trainer sees only a scalar, the environment should emit enough diagnostic information to reconstruct why that scalar was produced.

1. Write success and failure predicates before tuning reward weights.
2. Put every reward term into a named variable before summing.
3. Return the scalar reward to the trainer and the term breakdown in info.
4. Write unit tests for one success case, one task failure, and one time-limit truncation.
5. Audit evaluation tables for construct-matched success metrics, not only mean reward.

# Keep reward terms named before summing them into one scalar.
# The info dict preserves why the reward was assigned.
distance_to_goal = 0.04
collision = False
success = distance_to_goal <= 0.05 and not collision

reward_terms = {
    "success_bonus": 10.0 if success else 0.0,
    "distance_penalty": -distance_to_goal,
    "collision_penalty": -5.0 if collision else 0.0,
}
reward = sum(reward_terms.values())
terminated = success or collision
truncated = False
info = {"reward_terms": reward_terms, "success": success}

print(round(reward, 2), terminated, truncated)
print(info)

The expected output pairs a near-10 reward with terminated=True and a reward-term dictionary whose entries sum to the scalar reward. Readers should interpret that as a successful terminal transition whose score is explainable from named components rather than an opaque single number.

When an experiment about reward design and termination fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.