Section 15.6: Reward shaping and its hazards

A Careful Control Loop

Sparse rewards are hard for policy gradients because the rollout may contain thousands of actions before the first success signal. A robot might need to search, reach, align, grasp, lift, and place before seeing a final reward. Shaping adds smaller signals along the way, such as distance to a goal or uprightness.

The danger is objective mismatch. If a mobile robot receives reward for being close to a doorway, it may learn to hover at the doorway instead of passing through. If a manipulator receives reward for high gripper force, it may crush objects. The shaped reward must be treated as a training instrument, not as the final task definition.

Potential-based shaping is the safest standard form:

$$F(s,a,s')=\gamma\Phi(s')-\Phi(s).$$

Adding $F$ to the reward preserves the optimal policy under the usual discounted MDP assumptions because it changes trajectory returns by a telescoping potential term rather than changing which complete behavior is best.

Theory

A shaped reward should be decomposed in logs. Instead of storing only total reward, record task reward, progress shaping, energy penalties, contact penalties, safety penalties, and terminal success separately. This lets the team see whether PPO is improving the task or merely harvesting one component.

Embodied reward shaping also needs sensor humility. Distance-to-goal reward depends on pose estimation. Contact penalties depend on tactile or simulator contact models. Energy penalties depend on actuator measurements. If those signals are biased, the policy can learn the measurement artifact.

Worked Example

Code Fragment 1 compares a potential-based shaping bonus with a naive distance bonus. The potential term rewards progress between states; the naive term can keep paying the agent for merely being near the goal.

# Compare potential-based shaping with a naive proximity bonus.
# Potential shaping pays for progress, while proximity can reward loitering.
gamma = 0.99
distances = [5.0, 3.0, 2.0, 2.0]

def potential(distance: float) -> float:
    return -distance

for before, after in zip(distances, distances[1:]):
    potential_bonus = gamma * potential(after) - potential(before)
    proximity_bonus = 1.0 / (after + 1.0)
    print(before, "to", after, "potential:", round(potential_bonus, 3), "proximity:", round(proximity_bonus, 3))

The expected output makes the exploit visible in the last row. When the robot stops moving closer, the potential-based term collapses toward zero, but the proximity bonus keeps paying the same amount, which is exactly how a loitering behavior can become locally optimal.

The final row is the warning. A shaped reward that keeps paying without task progress creates a local strategy the optimizer can exploit. PPO will faithfully improve that proxy unless the evaluation artifact separates shaped reward from true success.

Practical Recipe

1. Define true task success before adding shaped reward components.
2. Prefer potential-based progress terms when a meaningful potential $\Phi(s)$ exists.
3. Log every reward component separately and plot component totals beside task success.
4. Run adversarial evaluations where the agent can exploit the shaping term without completing the task.
5. Keep shaped reward out of final claims unless task success, safety, and robustness metrics improve in the same run.

Reward shaping is a curriculum encoded as numbers. It can make hard exploration possible, but it can also teach a policy to satisfy the curriculum without graduating to the task. The safest workflow is to design shaping terms as hypotheses, then try to break them with targeted evaluations.

Potential-based shaping is valuable because it gives a formal condition for policy preservation, but the condition rests on assumptions. The potential must be a function of the Markov state used by the MDP, the discount convention must match the task, and the final evaluation must still use the real task objective. In partially observed embodied systems, the measured potential may be a noisy proxy for the true state potential.

Code Fragment 2 sketches the logging pattern that keeps shaping auditable. The total reward is useful for PPO, but the component record is what lets the team discover reward hacking.

1. Start with sparse task success and add one shaping term at a time.
2. For each term, write the exploit you expect the policy might discover.
3. Log shaped components, terminal success, safety flags, and videos in the same artifact.
4. Evaluate on scenarios where the shaping proxy and the task goal disagree.
5. Keep the smallest reward that trains reliably under the target perturbation panel.

# Record reward components separately from the scalar reward.
# Component logs reveal whether PPO is optimizing the task or exploiting a proxy.
from dataclasses import dataclass, asdict

@dataclass
class RewardComponents:
    task_success: float
    progress: float
    energy_penalty: float
    collision_penalty: float
    true_success: bool

    def as_row(self) -> dict[str, object]:
        return asdict(self)

components = RewardComponents(
    task_success=0.0,
    progress=0.8,
    energy_penalty=-0.1,
    collision_penalty=0.0,
    true_success=False,
)
total_reward = (
    components.task_success
    + components.progress
    + components.energy_penalty
    + components.collision_penalty
)
print(components.as_row())
print("total_reward:", total_reward)

The expected output shows why shaped reward alone is unsafe as a headline metric. The policy earns a positive total reward from progress despite true_success=False, so the correct interpretation is partial progress without task completion, not a solved episode.

When a shaped-reward policy fails, do not only lower the learning rate or change PPO parameters. First inspect which reward component dominated the advantages before failure. Then run a disagreement test: construct episodes where high shaped reward is possible without true success, and verify that the policy does not prefer that shortcut.