Section 33.1: What LLMs can and cannot do in embodied tasks

A Careful Control Loop

Read the figure as a boundary map. The LLM can propose task structure and language-level intent, but the embodied system still needs state estimation, affordance checks, typed action APIs, execution monitoring, and a verifier that can reject unsupported plans.

This section defines a clear contract for when an LLM should sit inside an embodied loop and when it should stay outside as an advisor or parser.

The practical question is not whether the LLM can describe the right action sequence, but whether that description can be turned into safe, low-latency, grounded behavior.

Theory

A convenient decomposition is $$\pi(a_t \mid h_t) = \kappa\bigl(\phi_\text{LLM}(x_t, m_t), \hat s_t\bigr),$$ where $\phi_\text{LLM}$ proposes a symbolic or programmatic plan from language context $x_t$ and memory $m_t$, while $\kappa$ is the grounded executor that consumes the proposal together with the current state estimate $\hat s_t$. The executor, not the LLM, owns physical validity.

This split explains both the promise and the limits. LLMs compress broad semantic priors into a small number of candidate subtasks or tool sequences. They do not directly measure friction, occlusion, latency, or actuator saturation. When papers claim strong embodied performance, the key question is how much the grounded stack contributes beyond the language model itself.

Worked Example

Code Fragment 1 implements the smallest planner boundary: the LLM proposes one symbolic subgoal, but the executor only accepts it if the required tool and state preconditions are satisfied.

# Accept an LLM proposal only when the grounded stack can execute it.
# The executor checks state and tool availability before acting.
# This boundary keeps semantic planning separate from physical validity.
proposal = {"step": "pick(red_mug)", "required_tool": "grasp"}
state = {"target_visible": True, "toolbox": {"grasp", "place"}}

can_execute = state["target_visible"] and proposal["required_tool"] in state["toolbox"]
decision = "execute" if can_execute else "replan"

print({"proposal": proposal["step"], "can_execute": can_execute, "decision": decision})

The expected output is an executable proposal whose semantic content and grounded feasibility agree. If `proposal` looked sensible but `can_execute` were `False`, the correct diagnosis would be a grounding or interface failure rather than a language-understanding success.

Practical Recipe

1. Write down which variables the LLM sees and which variables only the grounded stack sees.
2. Require every LLM proposal to map into a typed action or code object.
3. Attach a verifier to every proposal so textual plausibility never counts as success by itself.
4. Measure latency separately for planning, execution, and recovery.
5. Benchmark against strong non-LLM baselines on the same task contract before claiming an embodied gain.

A useful scientific discipline is to evaluate LLM contribution at the intervention boundary. Replace the LLM planner with a hand-written planner, a behavior tree, or a retrieval baseline while keeping execution fixed. If performance barely changes, the embodied value comes from the grounded stack, not from the language model.

This also reframes the hype around end-to-end embodied agents. The core question is not whether a model can emit an action token, but whether it can maintain a physically valid internal state, meet timing budgets, and recover from embodiment-specific failures. Most current systems still rely heavily on specialized modules for those responsibilities.

Code Fragment 2 saves a planner trace with the fields needed for real evaluation. The key idea is that every proposal is logged alongside its verifier result and latency, so semantic fluency cannot hide execution failure.

1. Record the prompt context or task card that generated the proposal.
2. Store the typed action, the verifier result, and the state preconditions checked before execution.
3. Measure planning latency separately from skill-execution latency.
4. Tag failures as semantic, state-estimation, tool-interface, or controller failures.
5. Compare planner variants on the same execution stack and episode set.

The expected output is a co-recorded control trace where the planner decision, the verifier evidence, and the latency measurement all point in the same direction. A useful negative case would keep the same structure but end with `result='blocked'` or `result='replan'`, which would let you localize the failure without replaying the whole episode.

If an embodied LLM system underperforms, first ask whether the LLM chose the wrong subgoal, whether the tool interface was incomplete, or whether the grounded stack could not realize an otherwise good plan. Those are very different scientific conclusions.