Section 26.2: The options framework

A Careful Control Loop

Why Hierarchy Matters

For The options framework, hierarchy separates timing, contact, recovery, and sequencing so a high-level planner can select skills without pretending every low-level policy is deterministic.

Options are the mathematical bridge between reinforcement learning and robot skills. They let a high-level policy choose temporally extended actions while the low-level policy handles many primitive steps before returning control.

Formal Contract

An option $\omega$ is a triple

$$\omega = (I,\, \pi,\, \beta),$$

where $I \subseteq \mathcal{S}$ is the initiation set (the states from which $\omega$ may be selected), $\pi: \mathcal{S} \to \Delta(\mathcal{A})$ is the intra-option policy (the primitive-action distribution the skill follows while running), and $\beta: \mathcal{S} \to [0,1]$ is the termination function (the probability of handing control back to the high-level policy in each state). The option executes for a random duration $\tau$ determined by $\beta$: at each step, it terminates with probability $\beta(s_t)$ and continues otherwise. The resulting process is a semi-MDP (SMDP) at the high level, because transitions take variable time.

The SMDP Q-function over options values starting state $s$ and selected option $\omega$:

$$Q_\Omega(s,\omega)=\mathbb{E}\left[\sum_{k=0}^{\tau-1}\gamma^k r_{t+k}+\gamma^\tau V_\Omega(s_{t+\tau})\mid s_t=s,\omega_t=\omega\right].$$

Here $\tau$ is the (random) option duration, the discounted sum accumulates primitive rewards over the option's execution, and $V_\Omega(s_{t+\tau})$ is the value at the state where the option terminates. The high-level policy selects from $\Omega$ (the set of all options) at every option boundary, not at every primitive time step. Use this tuple as an audit checklist when designing a skill: if any field of $(I, \pi, \beta)$ is implicit or missing, the option will fail silently at task boundaries.

Worked Implementation

Code Fragment 1 for The options framework should expose initiation, progress, termination, verification, and failure reporting before connecting the skill to ROS 2, BehaviorTree.CPP, Drake, or a learned policy.

# Simulate an option that moves through several primitive control ticks.
# The high-level controller sees one option outcome, not every inner action.
def execute_option(start_position, goal_position, max_steps=5):
    position = start_position
    trace = []
    for step in range(max_steps):
        delta = min(0.4, goal_position - position)
        position += delta
        trace.append(round(position, 2))
        if abs(goal_position - position) < 0.05:
            return "terminated", trace
    return "timeout", trace

status, trace = execute_option(0.0, 1.0)
print(status, trace)

The expected output trace shows exactly why the option is useful to a high-level planner: three internal control updates are compressed into one symbolic result. The final position reaches the goal within tolerance, so the option terminates cleanly instead of exposing each intermediate motor step.

Practical Recipe

1. Name each skill with a verb and object: navigate_to_station, grasp_handle, dock_drone, or change_lane.
2. Write preconditions, postconditions, safety guards, timeout, and recovery behavior before training a policy.
3. Represent sequencing as a finite-state graph, behavior tree, or task-and-motion plan so failures have explicit routes.
4. Use language as a planner only after commands are grounded into a typed skill library with affordance checks.
5. Evaluate composition, not only individual success. Many failures occur when two correct skills meet at a bad boundary.