Section 51.1: Closed- vs. open-world tasks

A closed-world benchmark is a tidy kitchen; deployment is the drawer where somebody put batteries, tape, and one mysterious screw.

A Junk Drawer Deployment

closed- vs. open-world tasks becomes useful when it is tied to a named interface, a replayable scenario, a failure diagnostic, and an artifact that records what changed in the action loop.

The key question is practical: What is fixed by the benchmark, what may change in deployment, and what evidence proves recovery rather than memorization?

Theory

For Closed- vs. open-world tasks, the practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Worked Example

Consider a mobile manipulator trained to pick known cups from known shelves. In the open world, the cup may be translucent, the shelf may move, and the instruction may refer to a new location.

# pip install gymnasium
import gymnasium as gym

env = gym.make("CartPole-v1")
obs, info = env.reset(seed=7)
for step in range(5):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    print(step, action, reward, terminated or truncated)

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

Closed- vs. open-world tasks becomes useful when it is tied to a closed-loop contract for Open-World and Novelty-Robust Embodiment. The contract names the participants, observations, action authority, timing budget, logging artifact, and recovery rule. Without that contract, a system can look capable in a notebook while failing the first time a partner delays, a person corrects it, or a deployment scene changes.

For Closed- vs. open-world tasks, separate the conceptual claim, the systems claim, and the evidence claim. A plausible mechanism, a clean interface, and a closed-loop result are different claims; the section should keep their evidence separate.

For Closed- vs. open-world tasks, the baseline and maintained-tool version should produce the same artifact schema and run on one task panel. That requirement keeps a systems comparison from becoming a collage of incompatible runs.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

When Closed- vs. open-world tasks fails, avoid labeling the whole method as weak. First assign the failure to perception, communication, human input, memory, planning, control, timing, data coverage, safety, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.

# Decide whether to act or abstain.
confidence = {"known_mug": 0.91, "unknown_object": 0.42}
threshold = 0.70

for item, score in confidence.items():
    decision = "act" if score >= threshold else "query_or_fallback"
    print(item, score, decision)

known_mug 0.91 act
unknown_object 0.42 query_or_fallback