Section 31.6: Human-agent interaction

A Careful Control Loop

For Human-agent interaction, read the figure as an interface check: identify the language input, grounding evidence, action representation, safety gate, and logged result before accepting the agent behavior described below.

This section explains how embodied agents should communicate uncertainty, accept corrections, and trade autonomy against user oversight during ongoing tasks.

The practical question is how much authority to give the robot before it must surface uncertainty or defer to the human.

Theory

Let $u_t$ denote a human input at time $t$, such as a command, correction, or approval. A shared-control policy can be written as $$a_t \sim \pi(a_t \mid h_t, x, u_{0:t}),$$ where the interaction history updates both task intent and trust calibration. The agent should not treat all human inputs equally: a correction signal often carries more control value than a new high-level command.

Interaction quality depends on observability in both directions. The robot must observe the user's intent, but the user must also observe enough of the robot's internal state to predict what it will do next. Explanations, preview actions, and confidence signals therefore become part of the control interface, not just user-interface decoration.

Worked Example

Code Fragment 1 shows a simple interaction gate that chooses between direct execution and confirmation. The policy uses both uncertainty and action risk, because even a confident proposal may deserve review if the consequence is expensive.

# Ask for confirmation when uncertainty or action risk is high.
# Human interaction is a control channel, not just a cosmetic interface.
# The gate should consider both confidence and consequence.
proposal_confidence = 0.58
action_risk = 0.72

need_confirmation = proposal_confidence < 0.7 or action_risk > 0.6
decision = "confirm" if need_confirmation else "execute"

print({"confidence": proposal_confidence, "risk": action_risk, "decision": decision})

Practical Recipe

1. Expose the agent's next intended action in a form the human can inspect quickly.
2. Define explicit interrupt and override channels for high-risk actions.
3. Treat corrections as informative state updates, not as episodic failure labels only.
4. Measure user effort: confirmations, overrides, and repair time are core metrics.
5. Tune the autonomy threshold on realistic tasks, because overly cautious systems become unusable.

Human-agent interaction is a good example of why embodied AI cannot be judged only by single-episode reward. The human is part of the loop, so the system should optimize for correction cost, legibility, and trust calibration in addition to nominal task success.

That perspective also changes how to think about demonstrations. A correction is not just a label saying 'wrong'; it is a structured intervention revealing where the human expected the robot's internal state to differ. Strong systems preserve that information for future planning and personalization.

Code Fragment 2 stores a minimal interaction event with proposal, human response, and final execution decision. That event is the unit you need for studying trust, intervention rate, and correction efficiency.

1. Log the robot proposal before the user responds.
2. Record whether the user approved, corrected, or overrode the action.
3. Update the task state or policy threshold after the interaction, not only after episode end.
4. Track intervention frequency together with success rate and completion time.
5. Replay interaction traces to measure whether the same misunderstanding recurs across tasks.

When interaction fails, separate perception or planning errors from interface design errors. A system may have good low-level control yet still be unusable because correction is too slow, too opaque, or too expensive for the human.