A Careful Control Loop
Language as a high-level controller treats action as a hierarchy rather than a flat stream of motor commands. A skill gives the planner a reusable temporal abstraction with an initiation condition, an internal policy, a termination rule, and a verification contract.
Why Hierarchy Matters
For Language as a high-level controller, hierarchy separates timing, contact, recovery, and sequencing so a high-level planner can select skills without pretending every low-level policy is deterministic.
Language can propose task decompositions, but it must be grounded into a typed skill library. The sentence 'inspect the shelf and bring the red cup' becomes executable only after perception resolves objects, affordance checks confirm feasible skills, and the planner maps words to verified actions.
For Language as a high-level controller, treat the skill as an interface: initiation set, internal controller, progress signal, termination rule, verifier, and recovery status must be explicit.
Formal Contract
For Language as a high-level controller, use the option tuple as an audit checklist: initiation states, internal policy, termination probability, and verifier must match the robot task.
$$p(\omega\mid u,o) \propto \exp(f_\theta(u,o,\omega))\,\mathbf{1}[\mathrm{affordance}(o,\omega)=1].$$
For Language as a high-level controller, map the option fields onto behavior trees, task graphs, finite-state machines, or task-and-motion planning nodes so start, act, stop, and verify remain inspectable.
Worked Implementation
Code Fragment 1 for Language as a high-level controller should expose initiation, progress, termination, verification, and failure reporting before connecting the skill to ROS 2, BehaviorTree.CPP, Drake, or a learned policy.
# Ground a language request into a typed skill sequence.
# The affordance check blocks skills that are not feasible in the current scene.
skills = {
"inspect": {"requires": "camera_ready"},
"navigate": {"requires": "map_ready"},
"grasp": {"requires": "object_reachable"},
}
scene = {"camera_ready": True, "map_ready": True, "object_reachable": False}
request = ["inspect", "navigate", "grasp"]
plan = []
for skill in request:
requirement = skills[skill]["requires"]
plan.append((skill, "allowed" if scene[requirement] else "blocked"))
print(plan)This expected output should be read as a grounded execution filter for language, not a language model answer by itself. The important outcome is that grasp stays blocked until the world state changes, which prevents fluent language from bypassing reachability constraints.
- Check whether the current state satisfies the skill initiation predicate.
- Execute the skill policy while monitoring progress, time, force, and perception confidence.
- Terminate when the skill succeeds, violates a safety guard, or reaches a timeout.
- Run a verifier that checks the postcondition in sensor space and task space.
- Return success, retry, fallback, or escalate to the high-level planner.
Practical Recipe
- Name each skill with a verb and object:
navigate_to_station,grasp_handle,dock_drone, orchange_lane. - Write preconditions, postconditions, safety guards, timeout, and recovery behavior before training a policy.
- Represent sequencing as a finite-state graph, behavior tree, or task-and-motion plan so failures have explicit routes.
- Use language as a planner only after commands are grounded into a typed skill library with affordance checks.
- Evaluate composition, not only individual success. Many failures occur when two correct skills meet at a bad boundary.
For Language as a high-level controller, use BehaviorTree.CPP, ROS 2 lifecycle nodes, Drake systems, or task-and-motion planning to handle scheduling and fallback while preserving explicit skill contracts.
For Language as a high-level controller, decompose the household command into navigation, inspection, reachability, grasp, carry, and handoff only if each subskill exposes a verifier and recovery route.
| Field | Question | Example For A Mobile Manipulator |
|---|---|---|
| Initiation | When may it start? | Object detected, arm clear, base within reach. |
| Policy | What controller runs? | Visual servoing plus impedance control. |
| Termination | When does it stop? | Grasp force stable for 0.5 seconds. |
| Verification | How is success proved? | Object pose follows gripper during lift. |
| Recovery | What happens after failure? | Open gripper, re-localize, retry from a safer pose. |
For Language as a high-level controller, test hierarchy failures caused by mismatched postconditions, hidden frames, stale perception, and planners treating probabilistic skills as deterministic.
For Language as a high-level controller, connect skill learning to VLA models and task-and-motion planning only when feasibility, verification, and recovery are represented for this body and scene.
For Language as a high-level controller, the test is whether initiation set, internal policy, termination rule, verifier, and recovery route can be written for the target robot skill.
Language as a high-level controller is useful when it makes the perception-action loop more reliable, not when it merely adds a more impressive model name.
Design a method-matched experiment for Language as a high-level controller. Specify the environment, observation schema, action interface, metric, and one perturbation that targets the section's core assumption.
What's Next
This section grounded language as a high-level controller in an explicit robot-data contract: observations, actions, demonstrations, evaluation splits, and failure labels. The next reading step is Section 26.5, where the same contract is carried into the next technique or chapter.
This paper formalizes options as temporally extended actions with initiation, policy, and termination conditions. It is the canonical reference for the chapter's skill hierarchy vocabulary.
Bacon, P. L., Harb, J., and Precup, D. (2017). The Option-Critic Architecture.
Option-Critic learns options end to end within reinforcement learning. It helps readers compare hand-specified skills with learned temporal abstractions.
Eysenbach, B. et al. (2018). Diversity is All You Need: Learning Skills Without a Reward Function.
DIAYN studies unsupervised skill discovery by maximizing distinguishable behaviors. It is useful for understanding when skills can be learned before a downstream task is specified.
Open X-Embodiment and RT-X Project Website.
Cross-embodiment datasets make skill reuse a practical question rather than only a theory topic. The project helps readers connect hierarchy to robot foundation models and shared behavior repertoires.
BehaviorTree.CPP Documentation.
Behavior trees are a production-friendly way to compose skills with fallback and monitoring logic. They complement learned policies by making high-level task decomposition explicit and inspectable.