Section 56.2: Spatial, episodic, and semantic memory

"The map knows where, the episode knows when, and the semantic store keeps insisting it knows why."

A Three-Drawer Memory Cabinet

Theory

The three memory families can be written as

$$\mathcal M_{\text{spatial}}=(V,E,\psi), \qquad \mathcal M_{\text{episodic}}=\{(o_{1:T},a_{1:T},r_{1:T},c)\}, \qquad \mathcal M_{\text{semantic}}=\{(k,v,\sigma,\eta)\}.$$

Spatial memory is often metric or topological. Episodic memory preserves trajectories and outcomes. Semantic memory stores abstractions or facts with confidence $\sigma$ and freshness $\eta$.

Operationally, these memory types usually sit in different toolchains. Spatial memory may live in `tf2`, occupancy maps, voxel stores, or scene graphs; episodic memory may be written into ROS bags, trajectory databases, or replay buffers; semantic memory may be indexed with FAISS, pgvector, or a knowledge graph. Separating the stores makes provenance and failure analysis tractable.

These memory types also differ in update cadence. Spatial memory may change every control cycle, episodic memory may be appended after each trajectory, and semantic memory may be updated only after aggregation or human verification. Conflating those cadences causes either excessive churn or stale abstractions.

Worked Example

A household robot searching for scissors may use spatial memory to remember drawer locations, episodic memory to recall where the scissors were last seen, and semantic memory to recall category-level storage priors.

def validate_memory_queries(payload: dict[str, object]) -> dict[str, object]:
    assert payload, "payload must not be empty"
    return payload

memory_queries = {
    "where_is_drawer_3": "spatial",
    "what_happened_last_time_i_opened_this_drawer": "episodic",
    "where_are_scissors_usually_stored": "semantic",
}
print(validate_memory_queries(memory_queries))

{'where_is_drawer_3': 'spatial', 'what_happened_last_time_i_opened_this_drawer': 'episodic', 'where_are_scissors_usually_stored': 'semantic'}

The expected output is simple, but the design implication is important. If a team cannot assign memory queries cleanly, the system usually lacks structure and later becomes hard to debug.

A useful evidence artifact here is a memory-routing ledger that records query text, chosen memory type, retrieved record id, freshness field, and downstream action change. That artifact makes it obvious when the robot answered a geometric question with a semantic prior or reused an old episode without checking whether the scene had changed.