Section 32.2: CLIP, SigLIP, DINOv2 representations

A Careful Control Loop

Inspect the diagram as an embedding audit. The useful question is whether CLIP, SigLIP, or DINOv2 features remain stable across viewpoint, lighting, distractors, and prompt wording while still preserving the action-relevant distinction.

Why These Representations Behave Differently

CLIP learns by contrasting matched image-text pairs against mismatched pairs in a batch. SigLIP keeps the same broad idea but replaces the softmax over the batch with independent sigmoid terms, which makes optimization behave better at smaller or more irregular batch sizes. DINOv2 is different again: it is self-supervised and language-free, so it often preserves patch-level visual structure even when no text prompt is available.

This means the "best" representation depends on the downstream interface. If the robot must retrieve the object referred to by language, CLIP or SigLIP is usually the starting point. If it needs dense correspondence, geometric consistency, or region matching before language enters, DINOv2 often becomes the stronger primitive.

Objectives And Their Consequences

CLIP-style training normalizes image and text embeddings, then learns them with a contrastive loss

where $S_{ij} = \tau \, f_I(I_i)^\top f_T(T_j)$ is the scaled similarity matrix over a batch. The softmax couples every pair through the batch normalization term. SigLIP instead applies a sigmoid loss to each pair, which reduces the dependence on giant globally synchronized batches. DINOv2 drops language entirely and learns invariant visual features through self-distillation, which is why it can be more reliable for patch similarity and visual tracking than for literal phrase grounding.

Worked Comparison

Code Fragment 1 uses toy embeddings to show the selection logic. The important idea is not the exact numbers, but the routing decision: semantic retrieval can prefer one candidate while dense visual consistency prefers another.

# Compare semantic similarity against dense-feature consistency for two objects.
# The first score approximates CLIP or SigLIP language alignment.
# The second score approximates DINOv2-style patch stability across views.
import numpy as np

objects = ["red_mug", "red_can"]
semantic = np.array([0.91, 0.73], dtype=float)
dense_consistency = np.array([0.58, 0.88], dtype=float)

language_best = objects[int(np.argmax(semantic))]
tracking_best = objects[int(np.argmax(dense_consistency))]

print({"language_best": language_best, "tracking_best": tracking_best})

The expected output is a deliberate disagreement: language retrieval selects red_mug, while dense visual consistency selects red_can. That split is useful because it proves the two representations are preserving different evidence; if both outputs were always identical, the section would not demonstrate why a fused embodied stack sometimes routes semantics and tracking through separate encoders.

When these signals disagree, the builder needs an interface rule. A common pattern is: let language choose the task-relevant object class, then let dense features maintain identity across viewpoint change or partial occlusion. This split mirrors the perception layering in 3D perception and scene representations.

Code Fragment 2 shows the maintained route for extracting embeddings from a DINOv2 checkpoint.

# Extract dense visual features from a maintained DINOv2 checkpoint.
# pip install transformers pillow torch
# The final hidden states can be pooled or kept per patch for tracking.
from PIL import Image
from transformers import AutoImageProcessor, AutoModel

model_id = "facebook/dinov2-base"
processor = AutoImageProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)

image = Image.open("tabletop_scene.png")
batch = processor(images=image, return_tensors="pt")
outputs = model(**batch)
print(outputs.last_hidden_state.shape)

The expected output is a tensor shape with batch size 1, 257 tokens, and 768 channels, which the reader should interpret as one global token plus a 16 by 16 grid of patch tokens. That patch structure is the reason DINOv2 is useful for correspondence and tracking: the output preserves spatially distributed evidence instead of only one scene-level classification vector.

Decision Table For Builders

The most robust embodied stacks keep semantic and geometric evidence separate long enough to debug them independently. A semantic encoder says whether the observation matches the instruction. A dense visual encoder says whether the same physical entity is still being tracked across time. Merging them too early makes failures hard to interpret, especially when a controller acts on stale or mismatched features.

There is also a systems cost question. CLIP-like encoders often win on simple promptable retrieval, but if the robot already pays for a tracking or mapping module, DINOv2 features may already exist in memory. Reusing those features can reduce latency and avoid redundant forward passes, which matters once the perception budget must fit a real control period.