Section 14.5: Why RL is hard in embodied systems (sample cost, reward, safety)

A Careful Control Loop

This section links back to Chapter 7: Control for AI Practitioners and Chapter 10: Environments with Gymnasium and PettingZoo, then prepares the policy-gradient work in Chapter 15: Policy Gradient Methods and PPO. The section explains why the formalism from the earlier sections becomes difficult once samples are physical, rewards are imperfect, and safety is nonnegotiable.

This section develops the technical contract for why RL is hard in embodied systems: sample cost, reward design, and safety constraints. First we connect these issues to the MDP/POMDP assumptions, then we express safety as a constrained objective, then we run a small numeric audit.

The key question is practical: what makes a high-return policy unacceptable when the learning process itself can damage hardware, surprise nearby people, or exploit a reward proxy?

Theory

Sample cost is the first obstacle. In a simulator, a failed episode may cost milliseconds. On hardware, it may cost a reset, a worn gripper, a human intervention, or a damaged object. This changes the acceptable exploration policy, the number of seeds, and the evaluation budget.

Reward design is the second obstacle. The reward $R(s,a,s')$ is a proxy for the task, not the task itself. A robot rewarded for moving a block near a target may learn to shove it violently, exploit perception blind spots, or end in unstable poses that score well for one frame. Sparse rewards delay credit assignment; dense rewards can teach the wrong shortcut.

Safety is the third obstacle. A constrained MDP writes the builder's intent more explicitly:

$$\max_\pi J_R(\pi)=\mathbb E_\pi\left[\sum_{t=0}^{\infty}\gamma^t r_{t+1}\right]\quad\text{subject to}\quad J_C(\pi)=\mathbb E_\pi\left[\sum_{t=0}^{\infty}\gamma^t c_{t+1}\right]\le d.$$

Here $r$ is task reward, $c$ is safety cost, and $d$ is the allowed discounted cost budget. The constraint matters because a policy can have excellent reward and still be unacceptable if it reaches that reward through collisions, excessive force, or unstable contacts.

Worked Example

Code Fragment 1 evaluates three candidate policies with the same reward and safety-cost definitions. The best policy by reward is not automatically acceptable because the safety budget is a separate constraint.

# Audit reward and safety cost for three embodied RL policies.
# A policy passes only if return is high and discounted cost stays within budget.
gamma = 0.9
safety_budget = 0.45
episodes = {
    "careful": {"rewards": [0.2, 0.5, 1.0], "costs": [0.0, 0.0, 0.1]},
    "fast": {"rewards": [0.4, 1.2, 1.8], "costs": [0.0, 0.4, 0.6]},
    "reckless": {"rewards": [1.0, 1.0, 2.0], "costs": [0.3, 0.5, 0.8]},
}

def discounted_sum(values):
    return sum((gamma ** t) * value for t, value in enumerate(values))

for name, trace in episodes.items():
    reward_return = discounted_sum(trace["rewards"])
    safety_cost = discounted_sum(trace["costs"])
    status = "pass" if safety_cost <= safety_budget else "reject"
    print(f"{name}: return={reward_return:.2f}, cost={safety_cost:.2f}, {status}")

The expected output ranks the policies two ways at once: by return and by discounted safety cost. The correct interpretation is that careful is the deployable winner because it stays under budget, while the higher-return policies fail the constraint and therefore do not count as acceptable solutions.

The result is a wins-only lesson for deployment: report the policy that satisfies the safety constraint and achieves the strongest validated return. Keep unsafe exploratory candidates in the experiment registry and diagnostics, not in the headline result.

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

The formal objective hides several engineering costs. The expectation in $J_R(\pi)$ assumes the agent can sample trajectories from the environment. In physical systems, each trajectory consumes calendar time, reset labor, battery cycles, hardware wear, and safety margin.

The reward and cost functions should be reviewed like interfaces. A reward that ignores force can favor damaging contact. A cost that triggers only after collision misses near-miss behavior. A reset procedure that changes object distribution can make evaluation easier than deployment.

A robust implementation treats safety and sample cost as first-class outputs. The experiment should be able to answer how many physical trials were used, how often a human intervened, how many constraint violations occurred, and which reward components drove the final policy.

1. Define reward and safety cost separately before training.
2. Set an allowed safety budget and a stop rule for violations.
3. Log resets and interventions with timestamps and causes.
4. Audit reward components for proxy shortcuts on saved videos or traces.
5. Report only policies that satisfy the safety budget on the shared evaluation panel.

When embodied RL fails, classify the failure before tuning. Sample-cost failures need better priors, models, demonstrations, or simulators. Reward failures need a repaired proxy. Safety failures need constraints, shields, or a different data-collection protocol.