Section 19.2: Intrinsic motivation, curiosity, count-based and novelty methods

A Careful Control Loop

It builds on reward specification in Chapter 18: Reward Design and Goal Specification, reuses partial observability from Chapter 2: The Agent-Environment Interface, and prepares transfer testing in Chapter 20: Sim-to-Real Transfer.

This section develops the technical contract for internal exploration rewards. Counts, pseudo-counts, prediction error, disagreement, and novelty all add an auxiliary reward signal, but they differ in what they treat as unknown.

The key question is practical: what feature representation receives the count or prediction error, and does that representation align with embodied progress such as new viewpoints, reachable states, object contacts, or goal-relevant affordances?

Theory

We can view the agent at time $t$ as receiving an observation $o_t$, encoding it as features $\phi(o_t)$, choosing an action $a_t$, and receiving both an external reward $r_t^{ext}$ and an intrinsic reward $r_t^{int}$. A count-based bonus often uses a form such as $r_t^{int}=\beta / \sqrt{N(\phi(o_t))}$, where $N$ is the visit count for the feature bin and $\beta$ sets the strength of the bonus.

Curiosity methods replace explicit counts with learnability. For example, a forward model predicts $\phi(o_{t+1})$ from $\phi(o_t)$ and $a_t$, then rewards prediction error when the next observation is not yet well modeled. The design danger is that unpredictable noise, moving shadows, or camera artifacts can look "interesting" even when they do not improve task competence.

Worked Example

Code Fragment 19.2.1 shows the count-based idea without hiding it inside a learner. A repeated feature receives a smaller bonus, so the agent has a measurable reason to leave the familiar hallway and test a new cell.

# Compute a count-based intrinsic bonus for visited feature bins.
# Repeated bins receive smaller rewards, so novelty decays locally.
from collections import Counter
import math

feature_trace = ["hall:0", "hall:1", "hall:1", "door:2", "door:2", "room:3"]
counts = Counter()
beta = 0.2

for feature in feature_trace:
    counts[feature] += 1
    bonus = beta / math.sqrt(counts[feature])
    print(feature, counts[feature], round(bonus, 3))

Expected output: repeated bins should show lower intrinsic reward than first-time bins. If the feature encoder maps every camera frame to a unique bin, this diagnostic would never decay and the agent would be paid for visual noise.

Practical Recipe

1. Choose the feature space for novelty before choosing the learning algorithm.
2. Plot intrinsic reward separately from external reward, action distance, and task progress.
3. Clamp or normalize the bonus so curiosity does not dominate the task reward for the whole run.
4. Record failures as structured cases: noisy feature, unreachable novelty, reward hacking, derailment, or evaluation mismatch.
5. Run at least one perturbation test that changes visual noise without changing the task state.

The idea in this section becomes useful when it is tied to a closed-loop bonus contract. In this chapter on Exploration in Embodied Worlds, the contract names the feature encoder, the count or prediction-error statistic, the action representation, the bonus scale, and the evaluation artifact. Without that contract, an agent can look curious while spending every rollout on unhelpful novelty.

The graduate-level habit is to separate three claims. The conceptual claim explains why the bonus should drive exploration. The representation claim explains what gets counted or predicted. The evidence claim records whether coverage, controllability, and task progress improve in the same run.

A robust implementation starts with a tiny, inspectable bonus trace and only then moves to a maintained learner. The baseline should log the feature key, visit count, intrinsic reward, external reward, action, and termination condition. The library version should produce the same artifact schema, so the comparison is a same-task comparison rather than a story assembled from separate experiments.

1. Write a one-paragraph bonus contract with feature key, count or prediction error, external reward, and failure fields.
2. Start with the smallest simulator or wrapper that exposes repeated and novel states clearly.
3. Run one deterministic smoke test and one nuisance-noise perturbation before scaling.
4. Save a single result artifact containing configuration, seed, rewards, counts, coverage traces, and failure labels.
5. Compare methods only when one script evaluates external reward, intrinsic reward, and coverage on the same task panel.

When an intrinsic reward method fails, avoid labeling the whole method as weak. First assign the failure to representation aliasing, uncontrollable noise, bonus domination, derailment, insufficient reset, sparse external reward, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause.