Section 15.3: Actor-critic and advantage estimation (GAE)

A Careful Control Loop

REINFORCE can assign a long delayed return to every action in a trajectory. That is unbiased, but it is noisy for embodied agents where a late slip, contact bounce, or tracking error can dominate the episode return. Actor-critic methods reduce this noise by learning $V_\phi(s)$, an estimate of how much return is expected from a state before the next action is chosen.

The actor update uses an advantage estimate:

$$A_t = Q(s_t,a_t)-V(s_t).$$

In practice, PPO often starts from the temporal-difference residual:

$$\delta_t=r_t+\gamma V(s_{t+1})-V(s_t),$$

then combines residuals with GAE:

$$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)}=\sum_{l=0}^{\infty}(\gamma\lambda)^l\delta_{t+l}.$$

Theory

The parameter $\lambda$ controls a bias-variance tradeoff. When $\lambda=0$, the advantage uses one-step TD evidence, which is low variance but depends heavily on critic accuracy. When $\lambda$ is near 1, the estimate resembles Monte Carlo return, which uses longer evidence but becomes noisier. PPO commonly uses values near 0.95 because that often balances delayed reward evidence with manageable variance.

For embodied control, this tradeoff is visible in time. A one-step residual may miss that a foot placement caused a stumble three frames later. A long-horizon return may blame that foot placement for an unrelated collision after a perception glitch. GAE gives the builder a knob for how far credit should travel backward through the rollout.

Worked Example

Code Fragment 1 computes GAE for a short rollout. Notice that the advantages are computed backward because later residuals influence earlier credit.

# Compute Generalized Advantage Estimation from rewards and value predictions.
# The backward recursion sends delayed evidence to earlier sampled actions.
rewards = [0.0, 0.2, 1.0, -0.1]
values = [0.3, 0.4, 0.6, 0.2, 0.0]
gamma = 0.99
lam = 0.95

advantages = []
gae = 0.0
for t in reversed(range(len(rewards))):
    delta = rewards[t] + gamma * values[t + 1] - values[t]
    gae = delta + gamma * lam * gae
    advantages.insert(0, round(gae, 3))

print("advantages:", advantages)

This trace gives the mental model. Step 3 has a negative advantage because the outcome was worse than the critic expected. Earlier steps remain positive because the later reward still supplies evidence that those actions helped set up a useful state.

Practical Recipe

1. Train the actor and critic on the same rollout batch so advantages match the behavior policy.
2. Bootstrap from $V(s_{t+1})$ when a rollout segment ends by truncation, but not when the episode truly terminates.
3. Normalize advantages per batch to prevent reward-scale changes from dominating the policy loss.
4. Track explained variance or value error so a broken critic does not silently corrupt the actor update.
5. Audit GAE under perturbations, because delayed physical failures change how far credit should travel backward.

The actor and critic optimize different targets from the same experience. The actor asks which sampled actions should become more likely. The critic asks which states predict future return. If these targets are mixed casually, a bug in value learning can masquerade as a policy improvement.

Embodied systems add one more complication: the critic often sees partial observations, not the full physical state. If the value function cannot infer hidden contact state, object slip, battery sag, or actuator temperature, its baseline will be noisy. That does not make actor-critic invalid, but it does mean the value diagnostics must be read alongside physical failure labels.

Code Fragment 2 shows the second piece most PPO implementations apply after GAE: advantage normalization. This is not part of the theorem; it is a practical stabilizer that keeps the policy loss scale consistent across batches.

1. Compute advantages before shuffling minibatches, using the rollout time order.
2. Normalize advantages after GAE, using the full batch mean and standard deviation.
3. Keep raw advantages in logs so reward-scale and critic failures remain visible.
4. Train the value function on returns compatible with the same bootstrap convention.
5. Plot value prediction, return target, and failure labels for several complete episodes.

# Normalize advantages after GAE so PPO sees a stable loss scale.
# Keep raw values for diagnostics because normalization can hide reward bugs.
advantages = [0.746, 0.691, 0.316, -0.300]
mean_advantage = sum(advantages) / len(advantages)
variance = sum((x - mean_advantage) ** 2 for x in advantages) / len(advantages)
std_advantage = variance ** 0.5

normalized = [(x - mean_advantage) / (std_advantage + 1e-8) for x in advantages]
print([round(x, 3) for x in normalized])

When actor-critic training fails, check critic health first. A value loss that falls while episode behavior worsens can mean the critic learned the wrong shortcut, such as predicting timeout length rather than task progress. A value loss that never falls can turn every advantage estimate into high-variance noise.