Section 12.4: Household and long-horizon: BEHAVIOR-1K / OmniGibson

A Careful Control Loop

What This Section Builds

This section makes household and long-horizon benchmarks operational. It focuses on tasks whose success depends on object states, room layouts, navigation, manipulation, and ordered subgoals rather than one isolated grasp.

The goal is to avoid a misleading binary score. A household policy can fail the final task while making meaningful predicate progress, or pass a task through a shortcut that ignores the intended behavior. The evaluation artifact must record both final success and subgoal evidence.

Theory

A long-horizon household benchmark is a predicate sequence over a simulated home. The policy must move through rooms, manipulate objects, change states, and satisfy a goal condition such as objects being placed, cleaned, opened, or arranged. The aggregate score should therefore include final success, predicate completion, path or action efficiency, and failure point.

The split must protect scene and object generalization. If train and test share the same layouts, object placements, or task templates, a policy may learn household shortcuts rather than robust behavior. If the same simulator seed appears in tuning and testing, the result is no longer a clean held-out household evaluation.

Worked Example

Code Fragment 1 turns long-horizon evaluation into predicate accounting. The key idea is that final success and partial progress should be co-computed from the same replay, not reconstructed from separate logs.

# Score household progress from a single ordered predicate trace.
# Final success and partial progress come from the same episode,
# which prevents mixing incompatible logs in one result table.
required_predicates = ("find_mug", "open_cabinet", "place_mug", "close_cabinet")
completed_predicates = ("find_mug", "open_cabinet", "place_mug")

completed = set(completed_predicates)
progress = sum(predicate in completed for predicate in required_predicates) / len(required_predicates)
final_success = all(predicate in completed for predicate in required_predicates)

print(f"progress={progress:.2f}, final_success={final_success}")

Practical Recipe

1. Define the household construct: navigation plus manipulation, object-state reasoning, task planning, recovery, or full long-horizon completion.
2. Freeze scene layouts, object instances, task templates, initial-state seeds, action horizon, and predicate definitions.
3. Report final success together with predicate progress, failure step, action count, and recovery events.
4. Stratify results by task length and scene novelty so short tasks do not hide long-horizon failures.
5. Save replays that show the object-state predicates changing over time.

Household and long-horizon benchmarks become useful when they preserve the causal story of the episode. The policy may find the object, move it, change its state, and still fail because a cabinet remains open or a target condition is not satisfied. A single final success number loses that structure.

The graduate-level habit is to co-compute final success, progress, and failure labels from one replay. A method that improves progress on hard scenes has a meaningful result even when final completion remains hard. A method that improves final success by exploiting easy templates needs a narrower claim.

A robust household evaluation starts by writing the predicate schema. The schema should list the subgoals, the state variables that make each predicate true, and the time step at which each predicate is evaluated. That schema becomes the bridge between long-horizon behavior and a reproducible metric.

1. Write the predicate schema and final success rule before running policies.
2. Freeze scene, object, task-template, and initial-state splits.
3. Run every method with the same horizon, action interface, and predicate evaluator.
4. Save per-predicate completion, final success, failure step, action count, and replay path.
5. Aggregate by task length and scene novelty, not only by a global mean.

Code Fragment 2 records a household evaluation result with both progress and final success. This lets the paper say exactly what improved.

# Record household evidence with scene split and predicate progress.
# Long-horizon results need the failure step and replay path because
# final success alone does not explain where the episode broke.
from dataclasses import dataclass, asdict

@dataclass
class HouseholdResult:
    suite: str
    scene_split: str
    task_template: str
    progress: float
    final_success: bool
    failure_step: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

result = HouseholdResult(
    suite="BEHAVIOR-1K",
    scene_split="unseen_homes",
    task_template="put_away_tableware",
    progress=0.75,
    final_success=False,
    failure_step="close_cabinet",
)
print(result.as_row())

Expected output: the printed result should expose scene split, task template, progress, final success, and failure step. Without those fields, a household benchmark hides the structure of the episode.

When a household experiment fails, inspect the first unsatisfied predicate. If the robot never found the object, test perception and navigation. If it found the object but could not change its state, test manipulation and physics. If it changed the object state but missed the final predicate, test the evaluator and task definition.