Section 24.4: Empirical data scaling laws in imitation learning

"The curve kept improving, but only after we stopped mixing validation splits like smoothie ingredients."

A Careful Curve Fitter

Power-Law Habit

A common empirical form is:

$$E(N) = A N^{-lpha} + E_{\infty},$$

where $E(N)$ is an error or failure rate after training on $N$ demonstrations, $lpha$ is the scaling exponent, and $E_{\infty}$ is the irreducible floor under the current setup. For success rates, researchers often fit error or regret rather than success directly because error has a natural decreasing trend.

Code Fragment 1 fits a line in log-log space to estimate a rough scaling exponent. The example uses small synthetic numbers so the arithmetic is transparent.

# Estimate a rough imitation-learning scaling exponent from matched runs.
# Every point must come from the same robot, split, metric, and training recipe.
import math

demo_counts = [100, 300, 1000, 3000]
failure_rates = [0.42, 0.30, 0.20, 0.14]

x = [math.log(n) for n in demo_counts]
y = [math.log(e) for e in failure_rates]
x_bar = sum(x) / len(x)
y_bar = sum(y) / len(y)
slope = sum((xi - x_bar) * (yi - y_bar) for xi, yi in zip(x, y)) / sum((xi - x_bar) ** 2 for xi in x)
alpha = -slope
print("alpha:", round(alpha, 2))

The expected output, alpha: 0.32, should be read as a local empirical summary, not a universal robotics constant. It says that under this exact synthetic panel, each multiplicative increase in demonstrations reduces failure at a rate consistent with a slope of about 0.32 in log-log space. A real paper should report confidence intervals, seeds, task-level scatter, and whether the fitted line still holds when the largest or smallest point is removed.

What A Scaling Point Must Contain

A single point on a robot scaling curve is a bundle of choices: dataset subset, task panel, model capacity, action representation, training compute, evaluation seed policy, and success metric. If any of those choices changes between points, the plot may still describe a useful engineering trend, but it no longer isolates data scale. Serious scaling studies therefore save one artifact per point with the subset manifest, model config, training logs, evaluation videos, and per-task outcomes.

There are two common variants. A data-only scaling study fixes model class and grows the number or diversity of demonstrations. A joint scaling study grows data and model capacity together. Both can be valid, but they answer different questions and should not be described with the same claim.