Section 32.5: Multimodal memory

A Careful Control Loop

Read the figure as a memory-indexing contract. Multimodal memory helps only when the retrieval key, stored evidence, temporal validity, and action consumer are explicit enough to prevent stale observations from steering the robot.

What Multimodal Memory Must Remember

A useful robot memory is not just a collection of captions or embeddings. It must bind semantics to geometry and time. A memory item might say: "mug_2, red, left of sink, pose in frame map, confidence 0.73, observed at 14.2 s, last revalidated at 15.1 s." Without pose and time, the memory cannot participate in planning. Without semantics, it cannot answer instruction-conditioned queries.

This makes multimodal memory a close relative of the world-state machinery in SLAM and map uncertainty. The difference is that Chapter 32 adds language labels and task-conditioned retrieval on top of geometric state.

Retrieval Score With Freshness And Uncertainty

Suppose each memory item $m_i$ contains a visual embedding $v_i$, a text summary embedding $t_i$, an age $\Delta t_i$, and an uncertainty penalty $u_i$. A practical retrieval score is

The first two terms reward semantic and visual match. The last two terms penalize staleness and uncertainty. This is the part many toy memory demos omit, but in robotics it is essential because an old perfect match can be less useful than a recent approximate match.

Worked Example

Code Fragment 1 makes that ranking rule explicit by combining visual match, text match, age, and uncertainty. This is the minimum interface needed before a vector database becomes genuinely useful for robotics.

# Rank memory entries by semantics, visuals, freshness, and uncertainty.
# The best entry should be useful now, not only historically descriptive.
# Age and uncertainty penalties keep stale evidence from dominating retrieval.
entries = [
    {"id": "mug_2", "visual": 0.82, "text": 0.77, "age": 0.4, "uncertainty": 0.08},
    {"id": "mug_2_old", "visual": 0.91, "text": 0.85, "age": 8.6, "uncertainty": 0.11},
]

def memory_score(item, alpha=0.45, beta=0.35, gamma=0.03, delta=0.40):
    return alpha * item["visual"] + beta * item["text"] - gamma * item["age"] - delta * item["uncertainty"]

ranked = sorted(((item["id"], round(memory_score(item), 3)) for item in entries), key=lambda pair: pair[1], reverse=True)
print(ranked)

The expected output is a ranked memory list where the fresher entry, mug_2, outranks the older but visually stronger one. That ordering is what the reader should check first: if mug_2_old still won, the scoring policy would be under-penalizing age and the memory system would behave more like static retrieval than embodied state estimation.

With this rule, a recent but slightly weaker observation can outrank an older perfect match. That behavior is often exactly what the planner needs when deciding whether to act immediately or revisit a location.

Code Fragment 2 shows the maintained pattern with a metadata-aware vector search.

# Query a vector memory with metadata filters for recency and source frame.
# pip install lancedb pyarrow
query_embedding = [0.12, -0.33, 0.48, 0.27]
filters = {"age_seconds_lt": 2.0, "frame_id": "map"}
mock_results = [
    {"object_id": "mug_2", "confidence": 0.74, "age_seconds": 1.1, "frame_id": "map"},
    {"object_id": "plate_1", "confidence": 0.63, "age_seconds": 3.4, "frame_id": "map"},
]
filtered = [row for row in mock_results if row["age_seconds"] < filters["age_seconds_lt"] and row["frame_id"] == filters["frame_id"]]
assert filtered
print({"top_object": filtered[0]["object_id"], "confidence": round(filtered[0]["confidence"], 2), "query_dim": len(query_embedding)})

The expected output is a single top retrieval whose object id and confidence survive both the vector similarity step and the metadata filter on age and frame. In other words, the system is not only finding a semantically similar memory, it is finding one that is still recent enough and frame-consistent enough to guide a new action.

Memory Schema For Embodied Systems

A durable schema usually needs at least these fields: object or region id, visual embedding, text summary, pose or frame reference, timestamp, confidence, source sensor, and invalidation rule. The invalidation rule is often overlooked, but it matters whenever a door can open, a person can move an object, or the robot itself can change viewpoint enough to break correspondence.

The most valuable memory audit is to replay one failure episode and ask which memory entry should have been ignored, updated, or invalidated. That question is usually more informative than asking which retrieval model had the best standalone benchmark score. In embodied systems, the policy around memory often matters as much as the embedding model inside it.