Section 32.3: Vision-language encoders and open-vocabulary detection

A Careful Control Loop

Read the figure as an open-vocabulary detection contract. The model may name objects outside a fixed label set, but the robot still needs boxes, masks, frame transforms, confidence calibration, and a rejection path for ambiguous detections.

What Open-Vocabulary Detection Actually Produces

A grounded detector does not return "the answer." It returns a set of candidate regions conditioned on language. A prompt such as "red mug" yields boxes or points that are semantically relevant, often before the system knows whether the object is graspable, visible enough for tracking, or safe to approach.

That distinction matters because detectors sit upstream of planning. They are proposal generators, much like how A* and sampling planners propose routes that still have to be checked against dynamics and safety constraints.

Scoring Boxes And Masks

A standard grounding stack scores each candidate box $b_i$ for text query $q$ with a detection score and a language-alignment score. A simple decision rule is

followed by thresholding, non-maximum suppression, and optional mask refinement. The multiplicative form is useful because it penalizes boxes that are visually dubious even if the language alignment is high. After a box is selected, a promptable segmentation model can turn it into a mask $m_i$, which is often what the robot actually needs for contact planning or free-space reasoning.

Viewed formally, this is a constrained optimization step under uncertainty: maximize the grounded score over proposals while enforcing overlap, visibility, and reachability constraints before the policy is allowed to act. The detector is therefore part of a state-space estimation pipeline, not merely a captioning front end.

You can also read the post-detection filter as a lightweight Bayes update with an implicit covariance budget: image-space evidence proposes hypotheses, geometric checks shrink the feasible set, and the final policy consumes only the survivors whose uncertainty has been reduced enough for action.

The final proposal set is therefore a filtered set $\mathcal B^\star = \operatorname{NMS}(\{b_i : \text{score}(b_i \mid q) \ge \tau\})$ that respects overlap suppression before any action module sees it. That algorithmic detail matters because embodied failures often come from duplicate or conflicting boxes surviving proposal selection, not from the language query alone.

For two proposals $b_i$ and $b_j$, the usual overlap test is $\operatorname{IoU}(b_i, b_j) = \frac{\lvert b_i \cap b_j \rvert}{\lvert b_i \cup b_j \rvert}$, and NMS suppresses the lower-scoring box when $\operatorname{IoU}(b_i, b_j) > \eta$. In robotics that threshold is not a cosmetic hyperparameter, because a too-low $\eta$ can erase small manipulable targets while a too-high $\eta$ can leave duplicate boxes that confuse the grasp selector.

Once a box $b^\star \in \mathcal B^\star$ survives, the embodied handoff is still incomplete until the system derives a world-frame action target, for example $\hat p = \Pi^{-1}(m^\star, D_t, T_{\text{camera}\rightarrow\text{world}})$ from the refined mask $m^\star$, current depth map $D_t$, and camera extrinsics. This makes the contract explicit: open-vocabulary detection proposes image-space regions, while action consumes geometry in a robot frame.

A minimal accept rule is therefore: execute only if $b^\star = \arg\max_{b \in \mathcal B^\star} \text{score}(b \mid q)$, the mask support exceeds a visibility threshold $\lvert m^\star \rvert \ge \kappa$, and the projected target satisfies clearance and reachability checks $c(\hat p) \le 0$. Stated this way, the detector is one stage in a constrained pipeline rather than a oracle.

Written compactly, the pipeline is $$ B_0 = \operatorname{Detect}(I_t, q), \quad s_i = \text{score}(b_i \mid q), \quad B^\star = \operatorname{NMS}(\{b_i \in B_0 : s_i \ge \tau\}), \quad m^\star = \operatorname{Seg}(I_t, b^\star), \quad \hat p = \Pi^{-1}(m^\star, D_t, T_{\text{camera}\rightarrow\text{world}}). $$ That sequence makes the module boundary explicit: language chooses proposals, segmentation sharpens support, and geometry converts the final mask into something a controller can actually use.

Worked Example

Code Fragment 1 compresses that ranking stage into a toy calculation. The goal is to make the selection rule concrete before we hide it behind a maintained GroundingDINO or Grounded-SAM pipeline.

# Rank grounded boxes by combining detector confidence and text alignment.
# This is the minimal decision rule before non-maximum suppression and masking.
# The robot should only pass top-ranked boxes to geometric verification.
import numpy as np

boxes = np.array(["left_box", "center_box", "right_box"])
det_scores = np.array([0.93, 0.74, 0.81], dtype=float)
text_scores = np.array([0.42, 0.89, 0.51], dtype=float)
joint_scores = det_scores * text_scores

best = int(np.argmax(joint_scores))
print({"selected_box": boxes[best], "joint_scores": joint_scores.round(3).tolist()})

The expected output is one selected region, center_box, together with three joint scores that show language evidence reshuffling the detector ranking. Reading the trace should make the logic legible: left_box was visually stronger on its own, but after text conditioning the center proposal becomes the correct candidate to pass into NMS, masking, and geometric verification.

Once the box is selected, the next question is whether the robot needs a rectangle or a pixel-accurate support region. For navigation or rough target selection, a box may be enough. For grasp planning, collision checking, or object-centric memory, a mask is usually much more useful.

Code Fragment 2 shows the maintained pattern at the point where most builders actually work.

# Ground a phrase to boxes, then refine the boxes into masks.
# pip install groundingdino-py segment-anything
# The detector proposes regions; the segmenter sharpens them for action.
image = load_image("tabletop_scene.png")
boxes, phrases = grounding_dino_predict(image, text_prompt="red mug", box_threshold=0.35)
masks = sam_refine_masks(image=image, boxes=boxes)

print({"num_boxes": len(boxes), "num_masks": len(masks), "top_phrase": phrases[0]})

The expected output is a small proposal set where the number of masks matches the surviving number of boxes and the top phrase remains tied to the user query. In a healthy GroundingDINO plus SAM-style pipeline, this tells the builder the semantic proposal stage and the geometric refinement stage stayed synchronized; if the counts diverged or the phrase changed, the handoff between detection and segmentation would need inspection.

Promptable Segmentation And 3D Follow-Through

A box is rarely the end of the story. If the robot must grasp, avoid, or remember the object, it often needs the mask to intersect with depth. The mask can be lifted into a point cloud, used to fit a 3D extent, or stored as a memory key for later re-identification. This is one of the clearest bridges from Chapter 32 to occupancy and neural scene representations.

The most useful diagnostic artifact for this section is a four-panel record: original image, top grounded boxes with scores, selected mask over depth, and the final action decision. That artifact lets you see whether the failure arose from phrase grounding, box ranking, mask quality, or geometric follow-through. It also stops teams from reporting open-vocabulary success on screenshots while the physical robot still grasps the wrong object.