Section 12.3: Lifelong and language-conditioned: LIBERO, CALVIN, Meta-World

A Careful Control Loop

What This Section Builds

This section makes lifelong and language-conditioned benchmarks operational. LIBERO tests knowledge transfer across language-conditioned manipulation suites, CALVIN tests long-horizon language-conditioned tabletop behavior, and Meta-World tests multi-task and meta-learning across a structured set of manipulation tasks.

The goal is to evaluate learning over distributions, not isolated episodes. That means freezing task order, language templates, held-out goals, adaptation budget, replay budget, seeds, and the rule for measuring forgetting.

Theory

A lifelong benchmark is a sequence of task distributions rather than one static test set. Let $S_{k,t}$ be success on task family $k$ after training stage $t$. A strong final average can hide catastrophic forgetting if earlier tasks lose success after later training, so the evidence artifact should include the whole task-by-stage matrix, not only the last column.

Language conditioning adds another split boundary. A policy can overfit to instruction templates, object names, or goal phrasing while failing to ground new combinations. The evaluation must say whether language, objects, spatial relations, goals, or task families are held out.

Worked Example

Code Fragment 1 computes a simple forgetting diagnostic. A method that improves the final task while damaging earlier tasks needs a different claim than a method that retains old skills and transfers to new ones.

# Measure forgetting from a task-by-stage success matrix.
# Rows are task families, columns are training stages after each
# new family has been introduced in the same order for every method.
success_by_stage = {
    "spatial": [0.68, 0.61, 0.55],
    "object": [0.00, 0.64, 0.58],
    "goal": [0.00, 0.00, 0.71],
}

forgetting = {}
for task, scores in success_by_stage.items():
    best_before_final = max(scores[:-1])
    if best_before_final == 0:
        forgetting[task] = "not_previously_trained"
    else:
        forgetting[task] = round(best_before_final - scores[-1], 2)

print(forgetting)

Practical Recipe

1. State the transfer claim: new objects, new spatial relations, new goals, new task families, longer language-conditioned chains, or faster adaptation.
2. Freeze task order, train/test task split, instruction templates, adaptation budget, replay budget, seeds, and stopping rule.
3. Report the full task-by-stage matrix for lifelong learning, plus final average success.
4. For Meta-World, separate multi-task performance from meta-learning adaptation to held-out tasks.
5. Label failures as language grounding error, task-order forgetting, adaptation overfit, object confusion, goal ambiguity, or controller failure.

Lifelong and language-conditioned benchmarks become useful when they expose the sequence of learning, not only the end state. A LIBERO result should show which knowledge shift was tested. A CALVIN result should show how language-conditioned chains were sampled and scored. A Meta-World result should say whether it evaluates multi-task training or adaptation to held-out tasks.

The graduate-level habit is to separate final competence from learning dynamics. Final success asks whether the policy can solve the last evaluation panel. Forward transfer asks whether earlier training helps later tasks. Backward transfer and forgetting ask whether later training preserves earlier skills. These are different constructs and must be co-computed from one task-by-stage artifact.

A robust transfer benchmark starts with a schedule file. The schedule file lists each training stage, task family, prompt set, replay source, adaptation budget, and evaluation panel. Every method reads the same schedule, which keeps the comparison tied to learning dynamics rather than run-specific choices.

1. Write a schedule with task order, prompt templates, train/test tasks, adaptation budget, replay budget, and seeds.
2. Evaluate after each stage on the same held-out panel, not only after final training.
3. Save the task-by-stage success matrix, final average, forward transfer, backward transfer, and forgetting.
4. Stratify results by task family or language condition so one easy family does not dominate the claim.
5. Keep prompt variants and demonstration access identical across methods.

Code Fragment 2 records a transfer schedule. This is the file you want reviewers to inspect before they trust a lifelong-learning number.

# Define one transfer schedule shared by every compared method.
# This schedule fixes task order, adaptation budget, replay budget,
# and held-out evaluation so final averages are construct-matched.
from dataclasses import dataclass, asdict

@dataclass
class TransferSchedule:
    suite: str
    task_order: tuple[str, ...]
    heldout_condition: str
    adaptation_steps: int
    replay_budget: int
    seeds: tuple[int, ...]

    def as_row(self) -> dict[str, object]:
        return asdict(self)

schedule = TransferSchedule(
    suite="LIBERO",
    task_order=("spatial", "object", "goal"),
    heldout_condition="new_language_object_combinations",
    adaptation_steps=0,
    replay_budget=200,
    seeds=(0, 1, 2),
)
print(schedule.as_row())

Expected output: the printed schedule should expose task order, held-out condition, adaptation budget, replay budget, and seeds. If two methods use different schedules, their final success values do not belong in one comparison.

When a lifelong or language-conditioned experiment fails, inspect the matrix before changing the model. A low final average may come from one hard task family, a prompt-grounding failure, forgetting of earlier tasks, insufficient adaptation budget, or a controller failure unrelated to transfer. The failure label should say which one.