Section 31.4: Object- and region-centric grounding

A Careful Control Loop

For Object- and region-centric grounding, read the figure as an interface check: identify the language input, grounding evidence, action representation, safety gate, and logged result before accepting the agent behavior described below.

This section explains why embodied systems often need both object-level and region-level grounding, especially when manipulation targets involve support surfaces, free space, or contact zones rather than only category labels.

The practical question is which representation best matches the next controller call: object id, pose, mask, affordance region, or continuous map cell set.

Theory

Let $z_i$ denote discrete object hypotheses and $R_j \subset \mathbb R^3$ denote grounded regions. A language-conditioned action interface should choose $$u = g(x, o_t) \in \{z_1, \ldots, z_n, R_1, \ldots, R_m\},$$ then pass either the discrete object handle or the continuous region geometry to the planner. The right choice depends on whether the downstream skill needs identity or geometry.

Object-centric grounding works well for pick, handover, or inspect actions where a discrete entity is the action subject. Region-centric grounding is better for 'wipe this spill,' 'place the bowl in the free space beside the plate,' or 'avoid the wet area,' because the relevant target is a spatial extent rather than a named object.

A more explicit control rule is to choose $\hat u = \arg\max_{u \in \{z_i, R_j\}} Q_{\text{skill}}(u \mid x, o_t)$, where the skill-specific value function changes with the consuming primitive. For grasping, $Q_{\text{skill}}$ typically rewards stable object identity and reachable pose, while for placement it rewards collision-free support area, clearance, and frame-consistent region geometry.

Worked Example

Code Fragment 1 compares an object-centric and region-centric interpretation of the same scene. The point is not the geometry itself, but the controller contract each interpretation enables.

# Compare object-centric and region-centric action targets.
# The object handle is enough for a simple pick, but placement needs a region.
# The chosen representation should match the action primitive downstream.
scene = {
    "target_object": "red_mug",
    "free_region_area_cm2": 128.0,
    "forbidden_region_area_cm2": 42.0,
}

pick_target = {"mode": "object", "handle": scene["target_object"]}
place_target = {"mode": "region", "free_area": scene["free_region_area_cm2"]}

print(pick_target)
print(place_target)

The expected output is two different target records from the same scene: one discrete handle for picking and one spatial summary for placing. If both lines came back as object handles, the pipeline would still be trapped at the noun level and the placement controller would be missing the free-region geometry it actually needs.

Practical Recipe

1. Map every skill primitive to the representation it expects before choosing a grounding model.
2. Use object ids for identity-sensitive tasks such as pick, inspect, and handover.
3. Use masks, surfaces, or free-space regions for placement, wiping, or collision avoidance.
4. Convert between object and region views explicitly, for example from mask to support surface.
5. Log the representation type in every evaluation trace so later comparisons stay construct matched.

Representation choice is one of the most under-reported design decisions in embodied language work. Papers often compare models while quietly changing what the downstream planner receives. An object token and a signed-distance field are not interchangeable interfaces, even if both originate from the same image and command.

The clean engineering pattern is to keep the language layer honest about this choice. If the command names a surface, the grounding module should output a surface representation. If the skill needs a free-space region, the pipeline should expose that region directly rather than pretending an object label is a sufficient proxy.

Code Fragment 2 records both representation type and frame. That detail matters because a region mask without its coordinate frame is not an actionable object for a planner.

1. Store whether the grounded target is an object, mask, point set, or free-space region.
2. Record the spatial frame and timestamp used to derive the target.
3. Pass discrete and continuous targets to different validator functions.
4. After execution, log whether the chosen representation was sufficient or needed refinement.
5. Compare systems only when they expose the same target representation to the same downstream skill.

The expected output is a region-centric record whose key fields are target_type='region', a named spatial frame, and a nonzero region-cell count. That combination tells the reader the grounding result is ready for a placement or navigation routine; if the frame were missing or the region size were zero, the output would be semantically plausible but not executable.

When region-centric tasks fail, inspect whether the wrong representation was chosen, whether the mask or surface was poor, or whether the planner consumed the geometry in the wrong frame. Treat these as distinct failure classes rather than as generic grounding errors.