Section 33.7: Memory, state tracking, and hallucination in physical tasks

A Careful Control Loop

Read the figure as a memory-validity audit. A planner may remember prior observations, but physical tasks require timestamps, scope, invalidation rules, and a check that retrieved state still matches the current scene.

This section shows how LLM memory should be paired with explicit state tracking so that past context helps planning without silently overriding new sensor evidence.

The practical question is which memories should live as symbolic facts, which should live as scene state, and how hallucinated memories should be caught before action.

Theory

Let memory items be facts $m_i = (f_i, c_i, t_i)$ with content, confidence, and timestamp. A planner should reason over a belief state $$b_t = p(s_t \mid o_{1:t}, a_{1:t-1}, m_{1:t}),$$ not over free-floating text summaries alone. New observations should update or erase memory items whose confidence is no longer justified.

Hallucination in embodied tasks often means one of three things: inventing an object or tool, asserting a stale state as current, or carrying a wrong relational fact across scene changes. The fix is rarely 'better prompting' alone. It is usually a better contract between memory, observation, and verification.

Worked Example

Code Fragment 1 stores two memories with different freshness and shows how the planner should gate them before use. The example demonstrates why timestamps belong in the memory schema.

# Reject stale world-state memory while keeping durable preference memory.
# Embodied memory should store freshness and source, not just text.
# This keeps old observations from masquerading as current state.
memory = [
    {"fact": "user_prefers_blue_mug", "age_s": 600, "durable": True},
    {"fact": "red_mug_is_on_counter", "age_s": 45, "durable": False},
]

usable = [m["fact"] for m in memory if m["durable"] or m["age_s"] < 10]
print(usable)

The expected output is a memory subset where durable user preferences survive but stale scene claims do not. The point is that embodied memory should grant planning authority only to facts whose lifespan matches the kind of fact they are, not to every retrieved sentence equally.

Practical Recipe

1. Store memory items with source, timestamp, confidence, and type.
2. Separate durable preferences from dynamic world-state facts.
3. Refresh or invalidate dynamic facts before high-consequence actions.
4. Never let retrieved text bypass a verifier when the action depends on current geometry.
5. Log contradictions between memory and observation as first-class events.

This section connects directly to classical filtering and SLAM. The novelty is that language memories and symbolic task facts must join the same belief-management discipline as geometric state. Otherwise the planner treats a ten-minute-old caption and a ten-millisecond-old sensor reading as equally authoritative.

That is also why hallucination should be decomposed. A model may hallucinate semantically, but many embodied 'hallucinations' are actually stale-state propagation errors. Better memory schemas, not bigger models, are often the right fix.

Code Fragment 2 stores a memory record with provenance and freshness. This is the minimum structure needed to talk coherently about embodied hallucination instead of merely complaining that the agent 'made something up.'

1. Tag each memory by type: preference, world state, task progress, or explanation.
2. Attach timestamps and evidence sources to every remembered fact.
3. Force memory retrieval to pass through a fact-validity gate before execution.
4. Record contradiction events when perception and memory disagree.
5. Evaluate memory systems on tasks with delayed execution and hidden state changes.

The expected output is a provenance-rich memory record that blocks direct action because the scene fact is too old. This is exactly the kind of trace you want before calling a behavior a hallucination, since the deeper mechanism is often stale world state rather than fabricated semantics.

When memory-rich agents fail, check whether the wrong fact was retrieved, whether the fact was stale, or whether the verifier failed to challenge it. Those paths lead to very different architectural fixes.