Section 16.4: Maximum-entropy RL

A Careful Control Loop

For Maximum-entropy RL, off-policy learning depends on replay semantics, environment API, target computation, and GPU-scale batching being fixed before comparison with policy-gradient methods.

Maximum-entropy RL addresses a failure that appears often in embodied control: a policy can become competent but too narrow. It succeeds when the world follows the training script, then fails when contact, lighting, object pose, or latency changes slightly.

The method changes the objective so the agent values both reward and controlled action diversity. The practical question is not whether randomness is good by itself. The question is how much stochasticity helps the robot discover and preserve useful alternatives without turning control into noise.

Theory

Standard RL maximizes expected return. Maximum-entropy RL augments return with the policy entropy at each state:

$$J(\pi)=\mathbb{E}_{\pi}\left[\sum_t r(s_t,a_t) + \alpha \mathcal{H}(\pi(\cdot|s_t))\right]$$

$\mathcal{H}$ is high when the policy spreads probability across multiple actions and low when it collapses onto one action. The temperature $\alpha$ sets the exchange rate between task reward and diversity. A high $\alpha$ encourages exploration and robustness; a low $\alpha$ makes the policy behave more greedily.

SAC implements this idea with soft value targets. In discrete notation, the soft value of a state can be written as:

$$V(s)=\alpha \log \sum_a \exp(Q(s,a)/\alpha)$$

This is a smooth version of $\max_a Q(s,a)$. As $\alpha$ becomes small, the highest action value dominates. As $\alpha$ grows, more actions contribute to the value.

Worked Example

Code Fragment 1 computes a soft value from three candidate action values. The higher temperature gives the lower-valued alternatives more influence, which is the numerical signature of preserving options.

# Compare hard max value with a maximum-entropy soft value.
# A larger temperature lets more actions contribute to the backup.
import math

q_values = [1.0, 0.7, 0.1]

def soft_value(values: list[float], temperature: float) -> float:
    scaled = [math.exp(value / temperature) for value in values]
    return temperature * math.log(sum(scaled))

for temperature in [0.2, 1.0]:
    print(f"alpha={temperature:.1f}", f"soft_value={soft_value(q_values, temperature):.3f}")
print(f"hard_max={max(q_values):.3f}")

For a robot, that difference means the policy can keep several near-good actions alive while it learns. The result can be better recovery behavior when the top action becomes unavailable after contact or a perception update.

Practical Recipe

1. Use maximum-entropy RL when exploration and recovery diversity matter.
2. Log entropy, temperature, action standard deviation, and action saturation.
3. Check reward scale before tuning $\alpha$, because reward magnitude changes the entropy tradeoff.
4. Evaluate deterministic and stochastic policy modes separately when the library supports both.
5. Stress-test whether entropy improves recovery under contact, delay, or occlusion shifts.

Maximum entropy is not a decoration on SAC. It is the reason the policy does not collapse immediately to whichever action currently has the highest estimated value. That matters when the critic is still wrong, the simulator is incomplete, or a robot needs recovery options after contact changes the state.

The engineering danger is reward-temperature mismatch. If rewards are scaled very large, entropy becomes negligible. If rewards are scaled very small, entropy can dominate. Good experiments report both the task metric and the entropy diagnostics that explain how the policy behaved.

A robust maximum-entropy implementation records the policy's diversity alongside task outcomes. Code Fragment 2 gives the minimum audit row needed to explain whether entropy was useful exploration or unsafe variance.

1. Log reward scale and all reward components.
2. Log entropy temperature and target entropy.
3. Log action mean, action standard deviation, and clipping frequency.
4. Report deterministic evaluation separately from stochastic evaluation.
5. Attach failure videos or traces for cases where entropy caused unsafe action spread.

# Build one audit record for maximum-entropy control.
# Entropy diagnostics explain whether diversity helped or harmed deployment.
from dataclasses import dataclass, asdict

@dataclass
class EntropyAuditRecord:
    reward_scale: float
    alpha: float
    target_entropy: float
    action_std: float
    clipped_fraction: float
    stochastic_eval_success: float
    deterministic_eval_success: float

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = EntropyAuditRecord(
    reward_scale=1.0,
    alpha=0.08,
    target_entropy=-2.0,
    action_std=0.31,
    clipped_fraction=0.04,
    stochastic_eval_success=0.82,
    deterministic_eval_success=0.79,
)
print(record.as_row())

When maximum-entropy control fails, separate three causes: the critic valued unsafe action diversity, the temperature kept entropy too high, or the reward scale made entropy too weak to matter. Then rerun with the same seed panel while plotting entropy, action spread, clipping, and recovery success.