Section 2.4: Rewards, goals, costs, constraints

"The robot maximized the reward exactly as written. That was the first problem."

A Reward Designer With New Gray Hair

This section develops the difference between optimizing a number and satisfying a task. Embodied systems operate around people, hardware, and physical limits. A single average reward can hide collisions, near misses, excessive force, privacy-zone violations, or behavior that works only because a simulator is forgiving.

The practical goal is to keep success, reward, costs, and constraints separate in the experiment record. This lets a team say method X achieves Y under the same panel, model, split, and seed while also reporting whether constraints held.

Theory

In reinforcement learning notation, the reward $r_t$ is often a scalar emitted after a transition. In robotics, the actual design space is broader. A goal might be "the cup is upright on the tray." A cost might be time, energy, jerk, or distance to humans. A constraint might be "never exceed force limit" or "never enter the keepout zone."

The important distinction is tradeability. Rewards and costs can be balanced when a tradeoff is acceptable. Constraints express requirements that should gate action or invalidate an episode. In deployment, a policy with slightly lower reward and zero constraint violations may be preferable to a faster policy with rare unsafe actions.

Worked Example

Code Fragment 2.4.1 scores two episodes. Both can complete the task, but only one satisfies the safety constraint.

# Section 2.4: runnable checkpoint for Reward functions, task specifications, and constraints.
# Keep the output small so the evidence record can be inspected directly.
def score_episode(success, seconds, collisions, entered_keepout):
    reward = 10.0 * float(success) - 0.05 * seconds - 2.0 * collisions
    costs = {"time_s": seconds, "collisions": collisions}
    constraints_ok = collisions == 0 and not entered_keepout
    return {"success": success, "reward": reward, "costs": costs, "constraints_ok": constraints_ok}

safe = score_episode(success=True, seconds=42, collisions=0, entered_keepout=False)
fast_unsafe = score_episode(success=True, seconds=20, collisions=1, entered_keepout=False)
print(safe)
print(fast_unsafe)

Expected output: two completed episodes with different safety status. The useful comparison is not only reward; it is reward plus the cost fields and constraint flag.

Practical Recipe

1. Write the goal in task language before writing a scalar reward.
2. Separate success, reward, cost vector, and constraint status in logs.
3. Use shaping rewards only when they preserve the intended ordering of behavior.
4. Add counterexample episodes that target reward hacking.
5. Report success with constraint violations, not success alone.

Reward design is safest when it is factorized before it is optimized. A goal describes the intended world condition. A reward provides learning or ranking pressure. A cost measures tradeoffs such as time, energy, jerk, distance to people, or interventions. A constraint names a boundary that should remain visible even when the reward improves.

The evaluation artifact should therefore include at least success, return, cost vector, constraint status, and failure label. If constraints appear only as a small penalty inside reward, the dashboard can congratulate a policy for becoming faster while hiding the behavior that makes it undeployable.

Build a reward audit that can catch reward hacking. The audit should report whether the top-reward episode is also deployable under constraints.

1. Write the task goal in ordinary language.
2. Define reward, each cost field, and each hard constraint separately.
3. Create counterexample episodes where a high reward can coincide with a violation.
4. Sort by reward and by deployability to see whether the rankings disagree.
5. Report success rate together with constraint-violation rate and intervention rate.

# Check whether the top reward episode is actually deployable.
episodes = [
    {"name": "safe_slow", "reward": 7.9, "success": True, "collisions": 0, "keepout": False},
    {"name": "fast_close_pass", "reward": 8.7, "success": True, "collisions": 0, "keepout": True},
    {"name": "fast_collision", "reward": 8.2, "success": True, "collisions": 1, "keepout": False},
]

def deployability(row: dict[str, object]) -> bool:
    return row["success"] and row["collisions"] == 0 and not row["keepout"]

top_reward = max(episodes, key=lambda row: row["reward"])
deployable = [row for row in episodes if deployability(row)]
print({"top_reward": top_reward["name"], "top_reward_deployable": deployability(top_reward)})
print({"best_deployable": max(deployable, key=lambda row: row["reward"])["name"]})

When a reward design fails, ask whether the goal was underspecified, a shaping term changed the intended ordering, a cost was hidden in the scalar, or a hard constraint was treated as a negotiable penalty. Fix the specification before tuning the learner.