Section 33.2: SayCan: affordance-grounded planning

A Careful Control Loop

Read the figure as the SayCan product rule in system form: language likelihood proposes what is useful, affordance likelihood estimates what is possible, and the robot acts only where both scores support an executable skill.

This section explains why affordance grounding is the natural antidote to free-text planning in robotics and why the combination is stronger than either source of evidence alone.

The practical question is how to combine semantic relevance and executability without letting one wash out the other.

Theory

SayCan scores each candidate skill $k$ with a language prior and an affordance value: $$k^* = \arg\max_k \; p_\text{LLM}(k \mid x, h_t) \cdot V_k(s_t).$$ The language term prefers semantically appropriate next steps, while the value term estimates whether the robot can execute that step successfully in the current state.

The multiplication matters. A skill with high semantic probability but near-zero affordance should be rejected, and a highly executable skill with no semantic relevance should not dominate just because it is easy. The method therefore depends on score calibration and on the quality of the candidate skill library.

Worked Example

Code Fragment 1 implements the core SayCan score on three skills. The example is tiny, but it makes the product structure visible and shows how the selected action can differ from the highest language score alone.

# Combine semantic plausibility with grounded affordance values.
# The best skill is not the one with the largest language score alone.
# Product scoring removes semantically attractive but infeasible actions.
skills = {
    "pick_sponge": {"p_llm": 0.55, "affordance": 0.92},
    "turn_on_sink": {"p_llm": 0.30, "affordance": 0.95},
    "wipe_spill": {"p_llm": 0.80, "affordance": 0.18},
}

combined = {name: round(v["p_llm"] * v["affordance"], 3) for name, v in skills.items()}
print(combined)
print(max(combined, key=combined.get))

The expected output is a ranking where the chosen skill is not merely the most semantically plausible sentence completion, but the action with the highest joint semantic and affordance score. Here `pick_sponge` wins because it is both relevant and executable in the present scene, while `wipe_spill` is semantically tempting but prematurely chosen.

Practical Recipe

1. Define a compact skill library whose actions expose clear preconditions and effects.
2. Generate only semantically plausible skill candidates rather than scoring the entire API surface.
3. Estimate affordance or value in the current state before execution, not from a stale scene snapshot.
4. Normalize or calibrate the two scores so one term does not dominate by scale alone.
5. Inspect failure cases where the right long-horizon plan starts with a low-probability semantic step.

The scientific subtlety in SayCan is calibration. The product formula is simple, but only meaningful if the two terms are roughly comparable in their interpretation. A miscalibrated value model can dominate the semantic term and reduce the planner to greedily choosing whichever skill is easiest right now.

The method also inherits the classic option-discovery problem from hierarchical RL. It can only select among skills it already knows. If the correct subtask is missing from the library, no amount of language fluency will recover it, which is why skill design and affordance learning remain central.

Code Fragment 2 stores the separate scores as an audit artifact rather than only the winning skill. This is the minimum needed to understand whether the semantic prior or the affordance estimator caused a bad decision.

1. Log the candidate skills and both scores for each decision point.
2. Keep the value-estimation state snapshot or seed so scores can be reproduced.
3. Store the chosen skill and the first rejected alternative for debugging.
4. Measure how often the affordance term changes the top language choice.
5. Benchmark with the same skill library and same execution stack when comparing alternatives.

The expected output is an audit tuple that preserves both factors of the SayCan product. If a future run selected the wrong skill with a high semantic score but a weak affordance score, the repair target would be calibration or candidate generation rather than the planner prompt alone.

If the chosen skill is poor, first check candidate generation, then affordance calibration, then library coverage. SayCan errors often come from what is missing from the candidate set, not only from how the final score is computed.