Section 1.1: Static prediction vs. embodied interaction

"A classifier answers a question once. An embodied agent answers, then inherits the consequences."

Section 1.1

The formal object changes

A static predictor learns a function $f(x)=y$. The input $x$ is drawn from a fixed distribution $\mathcal{D}$, the output is scored by a loss $\ell(f(x),y)$, and the example ends. Crucially, $\mathcal{D}$ does not depend on $f$: the test set is the same whether the model is good or bad.

An embodied agent acts inside a controlled Markov process. At step $t$ it receives an observation $o_t$, maintains a belief or internal state $b_t$, chooses an action $a_t \sim \pi(\cdot \mid b_t)$, and the world transitions $s_{t+1} \sim P(\cdot \mid s_t, a_t)$, emitting $o_{t+1}$. The unit of analysis is the trajectory $\tau = (o_0, a_0, o_1, a_1, \ldots, o_T)$, and the score is a functional of the policy,

$$J(\pi) = \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} r_t - \lambda \sum_{t=0}^{T} c_t\right],$$

where $r_t$ measures task progress, $c_t$ measures cost such as collision risk, energy, or recovery effort, and $\lambda$ encodes how much the evaluator penalizes unsafe or expensive behavior. The decisive term is the subscript on the expectation: $\tau \sim \pi$. The distribution of states the agent is judged on is produced by the agent. Improve the policy and you change the test set. This single coupling, absent from $f(x)=y$, is the source of distribution shift, compounding error, exploration cost, and the need for closed-loop evaluation.

Why error compounds: the horizon penalty

The practical consequence of $\tau \sim \pi$ is that imitation does not behave like supervised learning. Suppose a policy is trained to copy an expert and makes a mistake with probability at most $\epsilon$ on states drawn from the expert's distribution. In a static setting the expected number of mistakes over $T$ examples is $O(\epsilon T)$, linear and benign. In the closed loop, a single mistake moves the agent to a state the expert never visited, where the policy has no guarantee at all and is more likely to err again. Ross, Gordon, and Bagnell showed that this drives the expected cost of behavior cloning to $O(\epsilon T^2)$, quadratic in the horizon. The extra factor of $T$ is the price of the feedback coupling: errors are not independent, they steer the agent into regions where further errors are likely.

A minimal demonstration of compounding

The simulation below strips the phenomenon to its core. An agent is "on the expert's state distribution" until a per-step error knocks it off; once off, it rarely recovers because it was never trained on those states. We measure how off-distribution time grows with the horizon while holding the per-step error budget fixed. Linear growth would mean errors are independent; super-linear growth is the embodied penalty.

# Compounding of behavior-cloning error in a closed loop.
# Holds per-step error fixed and varies the horizon; off-distribution time grows super-linearly.
import numpy as np

def rollout_off_distribution_steps(horizon, step_error_prob, recover_prob, rng):
    on_track = True          # the agent starts on the expert's state distribution
    off_steps = 0
    for _ in range(horizon):
        if on_track:
            if rng.random() < step_error_prob:   # a wrong action leaves the expert's distribution
                on_track = False
        else:
            off_steps += 1                        # time spent in states the policy never trained on
            if rng.random() < recover_prob:       # recovery is rare: no supervision off-distribution
                on_track = True
    return off_steps

rng = np.random.default_rng(0)
print(f"{'horizon':>8} {'off-dist steps':>16} {'steps / horizon':>16}")
for horizon in (10, 40, 160, 640):
    off = np.mean([
        rollout_off_distribution_steps(horizon, step_error_prob=0.02, recover_prob=0.05, rng=rng)
        for _ in range(20000)
    ])
    print(f"{horizon:>8} {off:>16.2f} {off / horizon:>16.3f}")

Where point accuracy misleads

Because the state distribution is policy-induced, where an error occurs matters more than how often. An error near a reversible state is cheap; the same error rate concentrated near an irreversible transition (a collision, a dropped fragile object, a wheel over a ledge) is catastrophic. Two policies with identical aggregate accuracy can have opposite closed-loop value if one fails preferentially near irreversible states. The design response is to evaluate on trajectories, to weight cost by reversibility, and often to prefer a lower-confidence model with a reject or stop option over a higher-accuracy model with no abstention.