Section 31.5: Task planning from language; ambiguity and clarification

A Careful Control Loop

For Task planning from language; ambiguity and clarification, read the figure as an interface check: identify the language input, grounding evidence, action representation, safety gate, and logged result before accepting the agent behavior described below.

This section connects language planning to active information gathering by showing how an embodied agent should ask before acting when multiple interpretations lead to different risks or trajectories.

The practical question is not 'can the model generate a plan?' but 'should the agent trust the top plan without first reducing ambiguity?'

Theory

Let $m \in \mathcal M$ be a latent meaning of the instruction, and let $V(\pi, m)$ be the value of executing plan $\pi$ under that meaning. If the agent can ask a question $q$ with cost $c(q)$, the value of clarification is $$\operatorname{VoI}(q) = \mathbb E_{y \sim p(y \mid q)}\left[\max_\pi \mathbb E_{m \mid y, q} V(\pi, m)\right] - \max_\pi \mathbb E_m V(\pi, m) - c(q).$$ Ask when this quantity is positive.

In practice, the agent approximates this computation with confidence gaps, risk heuristics, or plan disagreement. The deeper lesson is that ambiguity should be represented in the planner's state rather than hidden inside the prompt. Otherwise the robot executes one interpretation while the human assumes another.

Worked Example

Code Fragment 1 computes a tiny expected-value test for whether to ask before acting. The numbers are synthetic, but the control logic is the same in household dialogue, warehouse dispatch, and mobile manipulation.

# Ask for clarification when plan value changes sharply across meanings.
# The cost of asking should be compared against the value of better execution.
# A small confidence gap does not matter unless it changes the chosen plan.
candidate_meanings = {
    "bring_red_mug": {"best_plan_value": 0.92},
    "bring_blue_mug": {"best_plan_value": 0.41},
}
ask_cost = 0.05
no_question_value = 0.5 * 0.92 + 0.5 * 0.41
after_question_value = max(0.92, 0.41)
voi = round(after_question_value - no_question_value - ask_cost, 2)

print({"no_question": round(no_question_value, 2), "after_question": after_question_value, "voi": voi})
print("ask" if voi > 0 else "act")

Practical Recipe

1. Maintain more than one candidate task object whenever the parse is not decisive.
2. Measure plan disagreement, risk difference, or verifier difference across those candidates.
3. Ask the smallest clarification question that collapses the uncertainty the most.
4. Treat the user's reply as a state update, then rerun grounding and planning.
5. Log the pre-question and post-question plan so ambiguity handling is auditable.

Clarification is best understood as a control action that changes the information state. It belongs in the same conceptual family as camera motion for better visibility or probing contact to reduce pose uncertainty. The agent spends time now to improve policy value later.

This framing also clarifies evaluation. A system that asks more questions is not automatically worse. It is worse only if those questions do not buy enough downstream value, such as safer execution, lower path length, or fewer catastrophic failures.

Code Fragment 2 stores the ambiguity state and the chosen clarification question in one artifact. The planner can then compare the original and revised plan without losing the reason the question was asked.

1. Store the top candidate meanings instead of only the winner.
2. Attach a question template to the specific slot that needs disambiguation.
3. Pause or gate dangerous actions until the reply is received or a timeout fires.
4. After the reply, re-run grounding and planning from the updated task object.
5. Audit whether clarification improved success, safety, or efficiency on the same episode set.

If the clarification loop underperforms, check whether ambiguity was detected too late, whether the wrong slot was queried, or whether the user reply failed to update the internal task object. These are distinct bugs with different remedies.