Section 51.2: Novel objects and instructions; changing environments

A mug with no handle is still a cup until the grasp planner files a complaint.

A Grasp Planner's Complaint

novel objects and instructions; changing environments becomes useful when it is tied to a named interface, a replayable scenario, a failure diagnostic, and an artifact that records what changed in the action loop.

The key question is practical: Which part is new: object appearance, object function, language, map, dynamics, or social context?

Theory

For Novel objects and instructions; changing environments, the practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Worked Example

Consider a robot asked to put the reusable bottle beside the compost bin in a kitchen it has never seen. The agent must infer object affordance, ground the instruction, update the map, and ask when ambiguity is high.

# pip install gymnasium
import gymnasium as gym

env = gym.make("CartPole-v1")
obs, info = env.reset(seed=7)
for step in range(5):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    print(step, action, reward, terminated or truncated)

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

Novel objects and instructions; changing environments becomes useful when it is tied to a closed-loop contract for Open-World and Novelty-Robust Embodiment. The contract names the participants, observations, action authority, timing budget, logging artifact, and recovery rule. Without that contract, a system can look capable in a notebook while failing the first time a partner delays, a person corrects it, or a deployment scene changes.

For Novel objects and instructions; changing environments, separate the conceptual claim, the systems claim, and the evidence claim. A plausible mechanism, a clean interface, and a closed-loop result are different claims; the section should keep their evidence separate.

For Novel objects and instructions; changing environments, the baseline and maintained-tool version should produce the same artifact schema and run on one task panel. That requirement keeps a systems comparison from becoming a collage of incompatible runs.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

When Novel objects and instructions; changing environments fails, avoid labeling the whole method as weak. First assign the failure to perception, communication, human input, memory, planning, control, timing, data coverage, safety, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.

# Choose a response based on novelty source.
novelty = {"type": "visual", "affordance_supported": False, "confidence": 0.48}

if novelty["confidence"] < 0.6 and not novelty["affordance_supported"]:
    action = "ask_or_collect_demo"
else:
    action = "reuse_policy"
print(novelty["type"], action)

visual ask_or_collect_demo