Section 33.5: ReKep: relational keypoint constraints

A Careful Control Loop

Read the figure as a relational-constraint contract. ReKep-style keypoints help when relations such as near, above, aligned, and graspable are grounded to tracked 3D points that survive motion and occlusion.

This section explains how language-guided agents can turn a command into a small set of geometric constraints over keypoints and then solve the resulting optimization problem.

The practical question is when relational keypoints are expressive enough to encode the task and when they become too brittle under clutter or contact.

Theory

Let keypoints be $k_1, \ldots, k_n$ and let a task be encoded by costs over relations among them. A trajectory optimizer can solve $$\tau^* = \arg\min_\tau \sum_j w_j c_j\bigl(k_{a_j}(\tau), k_{b_j}(\tau)\bigr),$$ where each $c_j$ measures a relation such as distance, alignment, or ordering implied by the language command.

This representation is attractive because it compresses a task into a few geometric relations that classical optimizers handle well. It is risky because errors in keypoint detection or object identity propagate directly into the objective, which can yield confident but wrong trajectories.

Worked Example

Code Fragment 1 evaluates a tiny relational cost between two keypoints. The specific numbers are simple, but they show how the language-derived objective becomes a concrete quantity an optimizer can minimize.

# Compute a simple relational keypoint cost for a grasp target.
# The task prefers the gripper keypoint to align closely with the mug handle.
# Small geometric costs are what the optimizer ultimately tries to drive down.
gripper = (0.42, 0.18)
handle = (0.47, 0.21)

cost = round(abs(gripper[0] - handle[0]) + abs(gripper[1] - handle[1]), 3)
print({"l1_alignment_cost": cost})

The expected output is a small geometric cost that operationalizes the verbal relation "align the gripper with the handle." A low value here means the relational abstraction has become numerically useful to an optimizer, while a high value would mean either the keypoints or the relation itself are not yet actionable.

Practical Recipe

1. Choose keypoints that correspond to task-relevant geometry such as handles, rims, hinges, or contact patches.
2. Translate the command into a small set of relational costs rather than a bag of verbal hints.
3. Estimate keypoint confidence and reject tasks whose geometry is too uncertain for safe optimization.
4. Use a planner or optimizer that can expose the final cost breakdown for debugging.
5. Compare keypoint-based and map-based formulations on the same task to see which abstraction is more stable.

ReKep is appealing because it lets language specify relations rather than every detail of a trajectory. That makes it a strong bridge between semantic tasking and numeric optimization, especially for manipulation tasks with a few dominant geometric constraints.

The limit is representational mismatch. Some tasks really are low dimensional in terms of keypoints. Others depend on extended surfaces, fluids, or occluded contacts. A strong engineer knows when the compact representation is a help and when it is a trap.

Code Fragment 2 saves the keypoint relation and its cost value as part of the experiment record. This is the right level of detail for deciding whether failure came from the vision front end or the optimizer.

1. Log keypoint identities, coordinates, confidence, and frame.
2. Store the relational costs that define the objective, not only the final trajectory.
3. Record whether the keypoints were visible, predicted, or carried from memory.
4. Compare optimizer output against the same keypoints under repeated seeds or perturbations.
5. Fallback to clarification or a denser representation when keypoint confidence collapses.

The expected output is a compact interface record: named keypoints, an explicit relational contract, the induced cost, and the downstream optimizer. This is the right evidence shape because it reveals whether a failure came from unstable keypoints, a poor relation choice, or a trajectory optimizer that could not satisfy a valid constraint.

If ReKep-style planning fails, inspect keypoint quality first, then relation design, then trajectory optimization. It is easy to blame the optimizer for what is actually a mis-specified or unstable geometric abstraction.