Section 16.1: Q-learning; deep Q-networks

A Careful Control Loop

For Q-learning; deep Q-networks, off-policy learning depends on replay semantics, environment API, target computation, and GPU-scale batching being fixed before comparison with policy-gradient methods.

Q-learning answers a specific embodied-agent problem: the robot often cannot wait for a full episode to learn whether a push, turn, or grasp was useful. It needs a local training signal after each transition, even when the reward is delayed and the next state is only partially observed.

The section develops that signal. We move from the tabular Bellman update, to the deep Q-network version that predicts action values from observations, to the stabilizers that make DQN usable when pixels, proprioception, and contact events keep changing the data distribution.

Theory

In tabular Q-learning, the agent updates one state-action entry after observing a transition $(s_t, a_t, r_t, s_{t+1})$. The target is the reward now plus the best discounted value the agent currently believes is available next:

$$y_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a')$$

The update moves the old estimate toward that target:

$$Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha \left[y_t - Q(s_t,a_t)\right]$$

The bracketed term is the TD error. In an embodied task, a positive TD error says the action produced a better consequence than expected, such as a door moving farther than predicted. A negative TD error says the action was overvalued, such as a grasp that looked good from the camera view but slipped under load.

DQN replaces the table with a neural network $Q_\theta(o,a)$, usually trained by minimizing the squared TD error over replayed transitions:

$$\mathcal{L}(\theta) = \left(r + \gamma \max_{a'} Q_{\theta^-}(o',a') - Q_\theta(o,a)\right)^2$$

$Q_\theta$ is the online network being trained. $Q_{\theta^-}$ is a slower target network that supplies the bootstrap value, which prevents the target from chasing every small online update.

It helps to name why DQN needs these stabilizers at all. Function approximation, bootstrapping, and off-policy learning together form the deadly triad: any one is fine, but all three at once can make value estimates diverge. Target networks and experience replay break the correlation between consecutive updates that drives that divergence, which is exactly why DQN keeps both.

Worked Example

Code Fragment 1 traces a single TD update with small numbers. The robot chooses a forward nudge, receives a small penalty for contact force, but the next state has one action that looks promising.

# Trace one Q-learning update with concrete values.
# The TD target combines immediate contact cost with the best next action value.
alpha = 0.25
gamma = 0.90
old_q = 0.40
reward = -0.10
next_action_values = [0.20, 0.80, 0.35]

target = reward + gamma * max(next_action_values)
td_error = target - old_q
new_q = old_q + alpha * td_error

print(f"target={target:.2f}")
print(f"td_error={td_error:.2f}")
print(f"updated_q={new_q:.2f}")

This numeric trace is the same logic DQN applies at scale. The difference is that DQN computes the current value and the target value with neural networks, then uses gradient descent to reduce the TD error over many replayed transitions.

Practical Recipe

1. Use Q-learning when the action set is discrete or can be discretized without hiding the control problem.
2. Define the reward so that near-term penalties, such as force spikes or collisions, do not disappear behind long-horizon success.
3. Track TD-error distributions, not only episode return. A widening TD-error tail often reveals bootstrapping instability.
4. Evaluate the greedy policy separately from the exploratory behavior policy.
5. Save per-transition logs with observation hashes, action ids, rewards, done flags, and target values.

The DQN design is a compromise between a clean Bellman equation and messy embodied data. Replay breaks short-range temporal correlation, the target network slows down the bootstrap target, and epsilon-greedy exploration keeps collecting non-greedy actions. Each stabilizer answers a specific failure: correlated frames, moving targets, and premature certainty.

For embodied agents, the hidden assumption is that the replay distribution contains enough coverage around the actions the greedy policy will later choose. If the robot learned mostly from safe, slow motions, the Q network may assign unreliable values to fast recovery actions that appear rarely but matter during deployment.

A robust DQN implementation starts with one inspectable Bellman update, then scales to replay and target networks. Code Fragment 2 records the fields that make a DQN run auditable: the online estimate, target estimate, TD error, and data-source label.

1. Log every transition as $(o,a,r,o',done)$ with episode id and seed.
2. Compute the target with a frozen or slowly updated target network.
3. Record the online Q value and the bootstrap component separately.
4. Plot TD-error quantiles by environment condition.
5. Compare greedy evaluation runs on one fixed perturbation panel.

# Build one audit record for a DQN target computation.
# Keeping target parts separate makes bootstrapping errors visible.
from dataclasses import dataclass, asdict

@dataclass
class DQNAuditRecord:
    transition_id: str
    action: str
    reward: float
    online_q: float
    target_q: float
    td_error: float
    source: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = DQNAuditRecord(
    transition_id="episode_014_step_032",
    action="nudge_forward",
    reward=-0.10,
    online_q=0.40,
    target_q=0.62,
    td_error=0.22,
    source="replay: low-friction block",
)
print(record.as_row())

When DQN fails, inspect whether the bad action came from perception error, reward misspecification, poor replay coverage, target-network lag, or overestimated bootstrap values. Then rerun the same evaluation panel while saving frames, selected actions, max-Q values, and TD errors. A failure with those fields becomes a diagnosis rather than a vague weak-model story.