Section 15.4: Trust regions; TRPO to PPO

A Careful Control Loop

A policy-gradient update can be too successful at following its own noisy estimate. One batch says a rare action looked good, the optimizer makes that action much more likely everywhere, and the next rollout discovers that the policy has stepped outside the region where the data were informative. In embodied agents, that can mean falls, collisions, or controllers driven into saturation.

TRPO frames the solution as a constrained optimization problem:

$$\max_\theta \mathbb{E}\left[\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)}\hat A_t\right]\quad\text{subject to}\quad \mathbb{E}\left[D_{\mathrm{KL}}(\pi_{\theta_{\mathrm{old}}}\Vert \pi_\theta)\right]\le \delta.$$

The probability ratio compares the new policy with the old behavior policy on the same sampled actions. The KL constraint says the new policy should not move too far from the old one in distribution space.

Theory

PPO replaces TRPO's constrained solver with clipping. Define the ratio

$$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)}.$$

The clipped surrogate objective is

$$L^{\mathrm{CLIP}}(\theta)=\mathbb{E}_t\left[\min\left(r_t(\theta)\hat A_t,\operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat A_t\right)\right].$$

If the advantage is positive, PPO stops rewarding the update once the new policy makes the action much more likely than the old policy did. If the advantage is negative, PPO stops rewarding the update once the new policy makes the action much less likely. The clip does not enforce a hard KL limit, but it removes incentive for many destructive updates.

The TRPO-to-PPO move is short to state. TRPO maximizes $\mathbb{E}\left[\frac{\pi_\theta}{\pi_{\mathrm{old}}}\hat A\right]$ subject to $D_{\mathrm{KL}} \le \delta$, solved with a conjugate-gradient step and a line search on a second-order trust-region model. PPO keeps the same surrogate ratio but replaces the hard constraint with the clipped objective above, so the same trust-region instinct now fits inside ordinary minibatch stochastic gradient descent.

Worked Example

Code Fragment 1 calculates the unclipped and clipped PPO terms for a few rollout samples. The clipped objective blocks extra credit when the probability ratio moves outside the trust band.

# Compute PPO clipped surrogate terms for sampled ratios and advantages.
# The min operation removes incentive for updates beyond the trust band.
ratios = [0.72, 0.93, 1.08, 1.31]
advantages = [0.6, -0.4, 0.8, 0.5]
epsilon = 0.2

for ratio, advantage in zip(ratios, advantages):
    clipped_ratio = min(max(ratio, 1 - epsilon), 1 + epsilon)
    unclipped = ratio * advantage
    clipped = clipped_ratio * advantage
    objective_term = min(unclipped, clipped)
    print(ratio, advantage, "clip:", clipped_ratio, "term:", round(objective_term, 3))

The expected output shows two different PPO behaviors. Ratios inside the trust band, such as 1.08, keep their natural objective term, while an oversized ratio like 1.31 is cut back to the clip limit, preventing one minibatch sample from driving an excessively large policy update.

The first row shows a different subtlety. A positive-advantage action became less likely, so clipping does not rescue it. The objective term remains low because the new policy moved against the learning signal.

Practical Recipe

1. Save old log probabilities during rollout and compute ratios from new log probabilities during training.
2. Use clip ranges such as 0.1 to 0.3 as starting points, then tune with KL, entropy, and return stability.
3. Stop or shrink updates when approximate KL exceeds the target by a large margin.
4. Track the fraction of samples clipped, because a high clip fraction means many gradients are pushing against the trust band.
5. Watch embodied safety metrics during updates, not only episodic return.

TRPO and PPO are best understood as responses to the same failure: on-policy data become stale quickly. TRPO solves the problem more formally with a constrained update based on KL divergence. PPO accepts a less exact constraint because the clipped loss is simple, scalable, and easy to combine with minibatch stochastic gradient descent.

For embodied systems, the important diagnostic is not only the scalar return. A KL spike can precede visible behavior collapse by one update: the rollout that caused the update still looked good, while the next policy executes a new distribution of motions. Logging approximate KL, clip fraction, entropy, and action standard deviation gives the team early warnings before a robot-level failure becomes expensive.

Code Fragment 2 shows a compact KL and clip-fraction diagnostic. These numbers belong next to return curves because they explain whether PPO improved behavior by a controlled update or by a risky jump.

1. Compute ratios as exp(new_log_prob - old_log_prob), not by dividing rounded probabilities.
2. Report approximate KL for each update epoch, not only at the end of training.
3. Stop the epoch loop early when KL exceeds the target threshold.
4. Pair every return plot with entropy, KL, and clip fraction.
5. Inspect videos or state traces from the update after the largest KL movement.

# Compute PPO diagnostics from old and new log probabilities.
# KL and clip fraction tell you whether the update stayed near the rollout policy.
import math

old_log_probs = [-0.9, -0.4, -1.2, -0.7]
new_log_probs = [-0.7, -0.5, -0.8, -0.2]
epsilon = 0.2

log_ratios = [new - old for old, new in zip(old_log_probs, new_log_probs)]
ratios = [math.exp(log_ratio) for log_ratio in log_ratios]
approx_kl = sum((ratio - 1.0) - log_ratio for ratio, log_ratio in zip(ratios, log_ratios)) / len(ratios)
clip_fraction = sum(abs(r - 1.0) > epsilon for r in ratios) / len(ratios)

print("ratios:", [round(r, 3) for r in ratios])
print("approx_kl:", round(approx_kl, 3))
print("clip_fraction:", round(clip_fraction, 2))

The expected output means three of the four sampled ratios are already outside the nominal PPO trust region, which is why the clip fraction rises to 0.75. A KL of 0.067 in such a tiny example should be read as a warning that additional epochs would likely over-update the policy.

When PPO collapses after a promising update, inspect the trust-region diagnostics before changing the reward. A high clip fraction suggests the learning rate, epoch count, or advantage scale is too aggressive. A low entropy trace suggests the policy lost exploration. A KL spike suggests old rollout data were overused.