Section 32.1: From image-text models to embodied perception

A Careful Control Loop

Read the figure as a grounding pipeline: image and language tokens are not yet robot state until the section explains how they become time-stamped facts, affordances, uncertainty, and a logged control decision.

From Similarity Scores to Actionable State

The core problem is that static image-text models are trained to score compatibility between an image and a phrase, while an embodied agent needs a state estimate that supports action. The state must include geometry, uncertainty, timing, and task context. A high image-text score alone does not tell the controller where to move or whether the observation is already stale.

This is why Chapter 27 on visual perception for action and Chapter 29 on localization matter here. VLM semantics usually enter the robot loop as one factor in a larger estimator, not as the estimator itself.

A Minimal Formal Contract

Let $I_t$ denote the current image, $q_t$ the language query for the task, and $s_t$ the latent scene state the robot actually needs. A pretrained image-text encoder gives embeddings $f_I(I_t)$ and $f_T(q_t)$ and a compatibility score

The score $\sigma_t$ is useful because it says whether the observation contains evidence for the queried concept. It is not yet enough for control. The embodied estimator still has to infer a belief over state,

where the belief depends on image history, previous actions, and task language. This link to belief state is what turns a static VLM into embodied perception rather than visual search.

Worked Numeric Example

Code Fragment 1 turns a few region-language similarities into a calibrated object-selection distribution. This mirrors the simplest embodied use case: the robot must choose which region deserves the next action or reobservation.

# Convert region-language similarities into a calibrated target distribution.
# The temperature term controls how aggressively the robot commits to one region.
# A low confidence gap should trigger another camera view instead of a grasp.
import numpy as np

region_names = np.array(["red_mug", "blue_bowl", "metal_sink"])
similarities = np.array([0.84, 0.79, 0.31], dtype=float)
temperature = 0.10

logits = similarities / temperature
probs = np.exp(logits - logits.max())
probs /= probs.sum()

best = int(np.argmax(probs))
margin = float(probs[best] - np.partition(probs, -2)[-2])

print({"target": region_names[best], "probabilities": probs.round(3).tolist(), "margin": round(margin, 3)})

The expected output is a normalized probability vector whose top class is red_mug and whose confidence margin stays explicitly visible. A builder should read this trace as "semantic evidence exists, but the gap is not yet huge," which is why the margin is stored as a control signal for reobservation rather than hidden inside a single winning label.

The probability gap is a small but important embodied quantity. If the top two regions are nearly tied, a cautious robot should gather another view or ask for a disambiguating instruction instead of treating the current winner as ground truth. This is the same uncertainty-sensitive design principle that appeared in state estimation and sensor fusion.

Code Fragment 2 shows that shortcut with the maintained CLIP interface.

# Use a maintained CLIP checkpoint to score one image against task phrases.
# pip install transformers pillow torch
# The processor handles resize, normalization, and tensor packing.
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

model_id = "openai/clip-vit-base-patch32"
processor = CLIPProcessor.from_pretrained(model_id)
model = CLIPModel.from_pretrained(model_id)

image = Image.open("tabletop_scene.png")
prompts = ["the red mug", "the blue bowl", "the metal sink"]
batch = processor(text=prompts, images=image, return_tensors="pt", padding=True)
logits = model(**batch).logits_per_image.softmax(dim=-1)
print(logits[0].tolist())

The expected output is one short probability list over the three task phrases, with the first phrase dominating but the second still nontrivial. In practice that means CLIP has semantic evidence for the mug, not a mathematically settled proof; if the top two numbers were nearly tied, the right next step would be a crop refinement, second view, or geometry check instead of immediate actuation.

Practical Embodiment Recipe

1. Write the task query in action language, for example "pick the red mug nearest the sink," not only class language such as "detect mugs."
2. Split perception into semantic evidence, geometry, and temporal freshness. Do not let a single score stand in for all three.
3. Use the VLM to rank candidate regions or hypotheses, then fuse that ranking with depth, pose, and reachability checks from camera and body frames.
4. Store the evidence artifact: image id, prompt, top regions, latency, confidence margin, final action, and failure label.
5. Evaluate on closed-loop success, recovery rate, and false-positive action rate, not only retrieval accuracy.

There are three progressively stronger uses of image-text models in robotics. The weakest use is captioning a frame and hoping a planner can infer everything else. The middle use is hypothesis ranking, where the VLM scores candidate regions, trajectories, or task interpretations that other modules generated. The strongest use is to make the score one observable term inside a structured state estimator whose outputs are explicitly consumed by planning and control.

The middle design is usually the best starting point for a real system because it respects modular boundaries. The detector or tracker proposes candidates, the VLM injects semantics, the geometry stack checks reachability, and the controller executes only when the evidence contract is satisfied. This also makes failure analysis cleaner: you can ask whether the error came from candidate generation, semantic ranking, calibration, or control timing.

When an image-text model appears to help, inspect one artifact that contains the scene image, prompts, region scores, confidence gap, depth estimate, chosen action, and episode result. Compare that artifact against a baseline policy on the same episodes. If the VLM raises retrieval scores but not task success, the missing variable is usually geometry, calibration, or latency rather than semantics.