Section 16.3: Continuous control: DDPG, TD3, SAC

A Careful Control Loop

For Continuous control: DDPG, TD3, SAC, off-policy learning depends on replay semantics, environment API, target computation, and GPU-scale batching being fixed before comparison with policy-gradient methods.

Continuous control creates a problem that tabular Q-learning cannot solve directly: a robot torque, steering angle, or gripper velocity can take infinitely many values. Taking a max over every possible action is no longer a small lookup.

DDPG, TD3, and SAC solve this by pairing a critic with an actor. The critic estimates value for a continuous action, while the actor proposes the action to evaluate or execute. This section explains how each method controls the bootstrapping error that appears when the critic and actor improve each other from imperfect off-policy data.

Theory

DDPG uses a deterministic actor $\mu_\phi(o)$ and a critic $Q_\theta(o,a)$. The actor is trained to choose actions that the critic values highly, while the critic is trained from replayed Bellman targets. This is efficient, but brittle: if the critic overestimates an action, the actor will move toward that action.

TD3 addresses that brittleness with three design choices. It trains two critics and uses the smaller target value, delays actor updates so the critic has time to improve, and adds small clipped noise to the target action so the critic cannot exploit a narrow spike in value. The target commonly has the form:

$$y = r + \gamma \min_i Q_{\theta_i^-}(o', \mu_{\phi^-}(o') + \epsilon)$$

SAC changes the objective by rewarding both task return and action entropy. The policy is stochastic, so the agent keeps useful diversity in its action choices:

$$J(\pi)=\mathbb{E}\left[\sum_t r(o_t,a_t) + \alpha \mathcal{H}(\pi(\cdot|o_t))\right]$$

The temperature $\alpha$ controls how much the policy values entropy. In contact-rich robotics, that entropy can help discover recovery actions that a deterministic actor would stop trying too early.

Worked Example

Code Fragment 1 computes a TD3-style target from two critics. The smaller critic value is used because overestimated value is more dangerous than underestimated value when the actor is trained to chase high values.

# Compute a TD3 target with clipped double critics.
# The smaller critic value limits overestimation before the actor sees it.
reward = 0.3
gamma = 0.98
critic_1_target = 1.40
critic_2_target = 0.90
target_policy_noise = 0.05
smoothed_action = 0.62 + target_policy_noise

bootstrap = min(critic_1_target, critic_2_target)
td3_target = reward + gamma * bootstrap

print(f"smoothed_action={smoothed_action:.2f}")
print(f"bootstrap={bootstrap:.2f}")
print(f"td3_target={td3_target:.2f}")

The expected output shows the TD3 target being anchored by the lower critic value, not the optimistic one. Readers should interpret the td3_target of 1.18 as a deliberately conservative bootstrap built from a slightly perturbed target action.

The same target logic maps cleanly to embodied control. For a torque-controlled arm, target smoothing says the critic should value a small neighborhood of torques, not a single fragile torque vector that only works in simulation.

Practical Recipe

1. Use DDPG only when a deterministic actor is acceptable and the task is well shaped.
2. Prefer TD3 when critic overestimation or narrow action spikes appear in evaluation.
3. Prefer SAC when exploration, recovery behavior, or multimodal actions matter.
4. Log action saturation at actuator bounds, since saturated actors can hide critic problems.
5. Evaluate under mass, friction, delay, and sensor-noise shifts with the same action limits.

DDPG, TD3, and SAC differ less in their environment interface than in their attitude toward critic error. DDPG trusts one critic and one deterministic actor. TD3 distrusts overestimation enough to train two critics. SAC distrusts premature certainty enough to optimize entropy alongside return.

That distinction matters in embodied work because physical action errors have asymmetric cost. An underestimated value may slow learning, but an overestimated high-torque action can break contact, drop an object, or leave the training distribution entirely.

A robust continuous-control implementation logs the actor's action and the critic's evidence for that action. Code Fragment 2 records the fields that separate deterministic control, clipped double-Q control, and entropy-regularized control.

1. Record action vectors before and after clipping or squashing.
2. Log critic disagreement for TD3 and SAC.
3. Log entropy or temperature for SAC.
4. Track actuator saturation and safety-envelope violations.
5. Compare all methods on one perturbation panel with the same action bounds.

# Build one audit record for a continuous-control action.
# The fields expose actor output, critic disagreement, and safety limits.
from dataclasses import dataclass, asdict

@dataclass
class ContinuousControlAudit:
    algorithm: str
    raw_action: float
    executed_action: float
    critic_1: float
    critic_2: float
    entropy: float
    saturated: bool

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = ContinuousControlAudit(
    algorithm="SAC",
    raw_action=1.18,
    executed_action=1.00,
    critic_1=2.4,
    critic_2=1.7,
    entropy=0.42,
    saturated=True,
)
print(record.as_row())

The expected output is an audit row that immediately exposes two continuous-control risks: the actor asked for more torque than the actuator allowed, and the critics disagree materially about the value of that command. That combination should trigger closer inspection before anyone treats the rollout return as robust evidence.

When a continuous-control method fails, first ask whether the actor left the safe action region, whether the critics disagreed, whether entropy collapsed, or whether replay lacked the new dynamics. Then rerun one perturbation while plotting action histograms and critic disagreement beside the episode video.