Section 31.3: Grounding language in perception; referring expressions

A Careful Control Loop

For Grounding language in perception; referring expressions, read the figure as an interface check: identify the language input, grounding evidence, action representation, safety gate, and logged result before accepting the agent behavior described below.

This section shows how embodied agents resolve names, attributes, and relations such as color, position, containment, and ownership into specific entities visible to the robot.

The practical question is how to combine visual evidence with relational language when several objects share the same category or attribute.

Theory

Given objects $z_1, \ldots, z_n$ extracted from perception and a referring expression $x$, the grounding problem is $$\hat z = \arg\max_i \; p(z_i \mid x, o_t) \propto p(x \mid z_i, o_t)\, p(z_i \mid o_t).$$ The perceptual prior says what objects are present; the language likelihood says which of those objects best matches the expression.

Relational language makes the problem harder because the referent depends on other objects. In 'the mug beside the kettle,' the target score depends on both the mug's own features and the probability that a nearby kettle exists and is correctly localized. Grounding therefore inherits every weakness of the detector and every ambiguity of the language model.

Worked Example

Code Fragment 1 scores three candidate objects against color, category, and spatial relation. It is deliberately tiny, but it exposes the same reasoning pattern used by larger grounding models and dialogue systems.

# Resolve a referring expression using attribute and relation evidence.
# Each candidate receives unary scores and a relation score to the kettle.
# The selected object is the one with the highest combined grounding score.
candidates = [
    {"id": "obj_1", "label": "mug", "color": "red", "near_kettle": True},
    {"id": "obj_2", "label": "mug", "color": "blue", "near_kettle": True},
    {"id": "obj_3", "label": "bowl", "color": "red", "near_kettle": False},
]

scores = {}
for obj in candidates:
    unary = 0.7 if obj["label"] == "mug" else 0.1
    color = 0.4 if obj["color"] == "red" else 0.0
    relation = 0.5 if obj["near_kettle"] else -0.2
    scores[obj["id"]] = round(unary + color + relation, 2)

print(scores)
print(max(scores, key=scores.get))

The expected output is a score table where obj_1 beats obj_2 by a modest relation-driven margin, followed by the winning referent id. That pattern tells the reader the grounding model used the kettle relation rather than only the noun mug; if obj_2 won instead, the likely failure is missing relational evidence rather than missing category evidence.

Practical Recipe

1. Detect or segment a candidate set before asking the language model to choose among them.
2. Score attributes and relations separately so you can diagnose which signal failed.
3. Keep the candidate set visible to the planner instead of passing only the winning object id.
4. When the top two candidates are close, trigger a clarification question or an active view change.
5. Re-ground after any action that changes visibility, object pose, or scene layout.

Referring expressions are a clean example of embodied partial observability. The agent may know the words but not the full scene, or it may see the scene but lack one relational anchor. Good systems therefore maintain uncertainty over the referent rather than forcing a premature point estimate.

This is also why closed-loop evaluation matters. A target selection model can have strong top-1 accuracy and still be a poor embodied component if it never signals uncertainty and therefore never triggers clarification or camera motion. The value of grounding lies in the final action outcome, not only in the static matching score.

Code Fragment 2 records the grounding result as an auditable object. The important fields are the chosen referent and the score margin over the runner-up, because that margin should drive clarification or active sensing.

1. Store the candidate list, the winning id, and the runner-up gap in one record.
2. Tie the winner to the current camera frame or map timestamp so stale groundings are detectable.
3. Route low-margin groundings to clarification or view-planning instead of execution.
4. Log whether the downstream action preserved the intended relation after grasp or motion.
5. Benchmark on scenes with distractors, occlusion, and viewpoint change, not only on clean still images.

The expected output is a compact grounding artifact with a winner, a runner-up, and a margin small enough to trigger clarify rather than execute. In a live system this exact trace is what separates a calibrated referential agent from an overconfident one: the same top prediction is preserved, but the action gate changes because the uncertainty is still operationally significant.

When grounding fails, separate detector failures, relation failures, stale-memory failures, and confidence-calibration failures. Different fixes apply to each: better proposals, explicit relation modeling, view planning, or threshold tuning.