Section 3.3: End-to-end learned policy pipeline

A Careful Control Loop

This section develops the technical contract for end-to-end learned policy pipeline into a usable mental model. First we define the object of study, then we connect it to the agent loop, then we test it with a compact implementation.

The key question in End-to-end learned policy pipeline is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

For End-to-end learned policy pipeline, the practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

An end-to-end policy is motivated by the handoff problem in modular stacks: if perception produces exactly the wrong abstraction, the planner never sees the information it needed. Instead of committing to intermediate symbols, the policy learns a direct map from observations and goals to actions:

$$a_t = \pi_\theta(o_{\le t}, g), \qquad \theta^\star = \arg\min_\theta \sum_{(o,g,a^\star)} \ell(\pi_\theta(o,g), a^\star).$$

The loss $\ell$ is often a regression loss for continuous actions, a cross-entropy loss for discrete action tokens, or a diffusion-style denoising loss for action chunks. The benefit is representation freedom: the model can keep visual, temporal, and language cues that a hand-written state estimator might discard. The cost is diagnostic opacity. When the policy fails, the builder must probe data coverage, action scaling, temporal context, embodiment mismatch, and distribution shift rather than opening a single broken module.

Worked Example

An end-to-end policy learns the map $a = \pi_\theta(o, g)$ directly from demonstrations, with no hand-written state in between. The example fits a minimal behavior-cloning policy by least squares, then makes the section's central point: when it fails, the first diagnostic is not "open a module" but "audit data coverage." We do that with a nearest-neighbor check against the training set.

import numpy as np
rng = np.random.default_rng(0)

# Demonstrations: observation = [obj_x, obj_y], goal g = 1 (pick).
# Expert action = unit vector from a fixed gripper origin to the object.
N = 200
O = rng.uniform(-1, 1, size=(N, 2))          # objects in a seen region
A = O / np.linalg.norm(O, axis=1, keepdims=True)   # expert reach direction

# Behavior cloning by least squares: a = O @ W  (the whole "training run").
W, *_ = np.linalg.lstsq(O, A, rcond=None)

def policy(o):
    return o @ W

def coverage(o, k=5):                          # distance to k nearest demos
    d = np.linalg.norm(O - o, axis=1)
    return np.sort(d)[:k].mean()

for label, o in [("in-distribution", np.array([0.3, -0.4])),
                 ("far OOD",        np.array([3.0,  3.0]))]:
    a = policy(o)
    err = np.linalg.norm(a - o / np.linalg.norm(o))
    print(f"{label:16s} cover={coverage(o):.2f} action_err={err:.2f}")

Expected output: the in-distribution query has small coverage distance and bounded action error; the far out-of-distribution query has a large coverage distance and a much larger error (the linear policy cannot extrapolate the normalized reach direction it never saw). That contrast is the diagnosis: the policy is not "broken," it is being asked about a region it never saw. This is why the first isolation test for an end-to-end policy is a coverage audit, not a module trace, because the architecture deliberately removed the modules you would otherwise inspect.

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

End-to-end learned policy pipeline becomes useful when it is tied to a closed-loop contract for how perception, estimation, planning, learning, and control are arranged into a system. The contract names the observation stream, the action representation, the timing budget, the safety boundary, and the result artifact. That is the bridge between a readable concept and a system a skeptical builder can test.

For End-to-end learned policy pipeline, separate the conceptual claim, the systems claim, and the evidence claim. A good explanation, a clean API, and one successful rollout are different kinds of evidence, and the section should keep them distinct.

For End-to-end learned policy pipeline, a robust implementation starts with one inspectable baseline whose artifact records observations, actions, units, timestamps, seeds, termination reasons, and the perturbation applied. The maintained-tool version is useful only if it preserves that schema and lets the comparison remain construct-matched.

1. Write a one-paragraph task contract with observation, action, success, failure, and safety fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save one artifact containing configuration, seed, metrics, traces, and failure labels.
5. Compare methods only when the same script evaluates the same panel, split, seed set, and metric.

When End-to-end learned policy pipeline fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.

For an end-to-end policy, the first isolation test is a nearest-neighbor audit of the training set: find the closest logged scenes, goals, and actions to the failed rollout. If the failed condition is absent, the diagnosis is coverage rather than architecture. If similar cases exist but the action scale, timing, or gripper convention differs, the diagnosis is representation alignment. If similar cases exist and conventions match, inspect the model's temporal window and action horizon before retraining.