Section 3.4: Hybrid and hierarchical architectures

A Careful Control Loop

This section develops the technical contract for hybrid and hierarchical architectures into a usable mental model. First we define the object of study, then we connect it to the agent loop, then we test it with a compact implementation.

The key question in Hybrid and hierarchical architectures is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

For Hybrid and hierarchical architectures, the practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Hybrid and hierarchical architectures exist because one controller rarely operates well at every time scale. A language-conditioned task policy may reason over minutes, a skill selector over seconds, and an impedance or velocity controller over milliseconds. The hierarchy separates "what should happen next" from "how this joint should move right now."

A useful formal model is the option view from hierarchical reinforcement learning. An option is $\omega = (I_\omega, \pi_\omega, \beta_\omega)$, where $I_\omega$ says when the skill may start, $\pi_\omega$ maps local observations to low-level actions, and $\beta_\omega$ says when the skill should terminate. The high-level policy chooses $\omega_t$, while the low-level policy emits actions until termination. This architecture works when skill boundaries are meaningful, termination is observable, and the high-level planner does not ask a low-level skill to satisfy an impossible precondition.

Worked Example

The option triple $\omega = (I_\omega, \pi_\omega, \beta_\omega)$ becomes concrete when each part is a function. The example runs a two-level controller on a reach-then-grasp task: a high-level policy picks an option, the option's low-level policy emits millisecond actions, and the option's termination function decides when control returns upward. The trace logs every boundary, which is exactly the evidence a hierarchy needs to be debuggable.

import numpy as np

# State: (gripper_x, gripper_closed). Goal: reach object at x*=1.0 then grasp.
X_TARGET, REACH_TOL, STEP = 1.0, 0.05, 0.25   # STEP = move per low-level tick

class Option:
    def __init__(self, name, can_start, low_policy, done):
        self.name, self.can_start = name, can_start
        self.low_policy, self.done = low_policy, done

reach = Option(
    "reach",
    can_start=lambda s: abs(s[0] - X_TARGET) > REACH_TOL,   # I_omega
    low_policy=lambda s: (np.sign(X_TARGET - s[0]) * STEP, 0),  # pi_omega
    done=lambda s: abs(s[0] - X_TARGET) <= REACH_TOL)        # beta_omega

grasp = Option(
    "grasp",
    can_start=lambda s: abs(s[0] - X_TARGET) <= REACH_TOL,   # precondition!
    low_policy=lambda s: (0.0, 1),
    done=lambda s: s[1] == 1)

def high_level(s, options):
    for opt in options:
        if opt.can_start(s):
            return opt
    return None

s = np.array([0.0, 0.0])
for t in range(20):
    opt = high_level(s, [grasp, reach])      # try grasp first, fall back to reach
    if opt is None:
        print("no applicable option"); break
    dx, close = opt.low_policy(s)
    s = np.array([s[0] + dx, max(s[1], close)])
    print(f"t={t:2d} option={opt.name:5s} x={s[0]:.2f} "
          f"closed={s[1]} terminate={opt.done(s)}")
    if opt.name == "grasp" and opt.done(s):
        print("task complete"); break

Expected output: the controller runs reach until the gripper is within tolerance, the high level then switches to grasp because its precondition is finally satisfied, and the task completes. The characteristic hierarchy bug is a precondition violation: delete the can_start guard on grasp and it will try to close on empty space at x=0. The logged precondition and termination fields are what let you assign that failure to the right level instead of blaming the whole stack.

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

Hybrid and hierarchical architectures becomes useful when it is tied to a closed-loop contract for how perception, estimation, planning, learning, and control are arranged into a system. The contract names the observation stream, the action representation, the timing budget, the safety boundary, and the result artifact. That is the bridge between a readable concept and a system a skeptical builder can test.

For Hybrid and hierarchical architectures, separate the conceptual claim, the systems claim, and the evidence claim. A good explanation, a clean API, and one successful rollout are different kinds of evidence, and the section should keep them distinct.

For Hybrid and hierarchical architectures, a robust implementation starts with one inspectable baseline whose artifact records observations, actions, units, timestamps, seeds, termination reasons, and the perturbation applied. The maintained-tool version is useful only if it preserves that schema and lets the comparison remain construct-matched.

1. Write a one-paragraph task contract with observation, action, success, failure, and safety fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save one artifact containing configuration, seed, metrics, traces, and failure labels.
5. Compare methods only when the same script evaluates the same panel, split, seed set, and metric.

When Hybrid and hierarchical architectures fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.

The characteristic hierarchy failure is a mismatch between levels. The high-level planner may select "open drawer" when the gripper is not aligned, or the low-level skill may report success after moving the handle without opening the drawer. Diagnose this by logging skill preconditions, termination causes, and residual state error at every boundary. A hierarchy is healthy when each layer can say why it accepted the task and why it stopped.