Section 32.4: Visual question answering and scene description in environments

A Careful Control Loop

Read the figure as a question-answering safety check. A scene description matters only when the answer is grounded to visible evidence, bounded by uncertainty, and routed to a planner that can refuse unsupported commands.

Captioning Is Not State Estimation

A caption like "A mug sits near the sink" is useful context, but an embodied agent typically needs a structured assertion such as {target: mug_2, relation: left_of_sink, confidence: 0.68, source_frame: 1842}. The first form is descriptive prose. The second form can be fused with memory, checked against geometry, and invalidated when the scene changes.

This is why VQA belongs near language-guided agent interfaces and belief updating. The model's answer must become evidence, not just commentary.

Question Answering As Conditional Inference

Formally, VQA asks for an answer $z_t$ conditioned on image $I_t$ and query $q_t$,

For embodied use, the answer should usually be factored into a structured state proposal $y_t$ plus an uncertainty score $u_t$. A simple selective-answering rule is

The threshold $\tau$ is not cosmetic. It encodes a system decision about when the robot should ask another question, change viewpoint, or escalate to a safer fallback.

Worked Example

Code Fragment 1 converts candidate VQA answers into a structured state field with abstention. The point is not to build a full model in a compact example, but to show the discipline embodied systems need at the interface.

# Convert VQA candidates into an action-ready state field with abstention.
# The planner consumes a typed answer only if the confidence clears a threshold.
# Otherwise the robot should reobserve or ask a narrower question.
answers = [
    {"value": "left_of_sink", "prob": 0.62},
    {"value": "on_counter", "prob": 0.27},
    {"value": "unknown", "prob": 0.11},
]
threshold = 0.70
best = max(answers, key=lambda item: item["prob"])

state_update = {
    "relation": best["value"] if best["prob"] >= threshold else "ABSTAIN",
    "confidence": round(best["prob"], 2),
}
print(state_update)

The expected output is an abstaining state update rather than a forced relation label, because the confidence stays below the 0.70 gate. That behavior is the point of the example: a good embodied VQA interface should surface uncertainty in a way a planner can act on, not quietly convert every plausible answer into a brittle world-state assertion.

Notice what changed: the model's job was not to produce eloquence, but to update a typed relation. This is the same transition from language to control-relevant structure that appears in Chapter 33 on planners and controllers.

Code Fragment 2 shows the maintained pattern with a multimodal generation interface.

# Ask a multimodal model for a structured relation answer.
# pip install transformers pillow torch
# The response should be parsed into typed fields before planning uses it.
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "google/paligemma-3b-mix-224"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id)

image = Image.open("kitchen_scene.png")
prompt = "Answer with JSON: relation of the red mug to the sink."
batch = processor(images=image, text=prompt, return_tensors="pt")
output_ids = model.generate(**batch, max_new_tokens=32)
print(processor.batch_decode(output_ids, skip_special_tokens=True)[0])

The expected output is one short JSON-style answer with both a typed relation and a confidence that clears the local action gate. A reader should interpret this as "the model produced something the planner could plausibly consume," not as proof that the relation is globally true forever, which is why timestamping and later invalidation still matter.

Latency And Resolution Tradeoffs

VQA becomes fragile in a real robot loop when the model is slow enough that the scene changes before the answer arrives. A larger resolution or larger model may improve descriptive accuracy while degrading action accuracy because the answer is late. The right metric is often answer usefulness under a control deadline, not raw answer quality.

The best chapter-level diagnostic for this topic is to log the raw image, the question, the free-form answer, the structured parse, the confidence score, and the downstream action decision together. That artifact reveals where the value really came from. Many systems appear to succeed because a human evaluator likes the answer wording even when the structured state is still too ambiguous for planning.