Section 12.5: Navigation and social: Habitat 3.0, AI2-THOR / ProcTHOR

A Careful Control Loop

What This Section Builds

This section makes navigation and social benchmarks operational. It distinguishes pure navigation success from efficient navigation, rearrangement progress, and social behavior such as following a humanoid while maintaining safe distance.

The goal is to keep path metrics, scene splits, generated-house seeds, and human-interaction rules attached to the result. Without those fields, a leaderboard row can hide whether a policy learned navigation, memorized layouts, or benefited from easier generated homes.

Theory

Navigation benchmarks often report success and path efficiency together. If the shortest path length is $L$ and the agent path length is $P$, success weighted by path length is $S \cdot L / \max(P, L)$, where $S$ is 1 for success and 0 for failure. This prevents a policy from getting full credit for eventually reaching the goal through an inefficient route.

Social navigation adds another constraint: the robot should achieve its goal without crowding, blocking, or colliding with the humanoid or human collaborator. Habitat 3.0 makes this explicit through collaborative tasks such as social navigation and social rearrangement, while AI2-THOR and ProcTHOR emphasize interactive indoor environments and procedurally generated houses.

Worked Example

Code Fragment 1 computes success weighted by path length for three episodes. The same pass also keeps collisions visible, because a socially unsafe route should not be celebrated as an efficient route.

# Compute navigation efficiency from the same episode table as safety.
# Success weighted by path length rewards reaching the goal efficiently,
# while collision counts keep unsafe routes visible.
episodes = [
    {"success": 1, "shortest": 6.0, "actual": 7.5, "collisions": 0},
    {"success": 1, "shortest": 5.0, "actual": 12.0, "collisions": 2},
    {"success": 0, "shortest": 8.0, "actual": 10.0, "collisions": 1},
]

spl_values = [
    ep["success"] * ep["shortest"] / max(ep["actual"], ep["shortest"])
    for ep in episodes
]
mean_spl = sum(spl_values) / len(spl_values)
total_collisions = sum(ep["collisions"] for ep in episodes)

print(f"mean_spl={mean_spl:.2f}, total_collisions={total_collisions}")

Practical Recipe

1. Choose the construct: point navigation, object navigation, rearrangement, social navigation, or social rearrangement.
2. Freeze scene split, generated-house seed list, target sampling, sensor suite, action horizon, and path-length normalization.
3. Report success, path efficiency, collisions, timeout rate, and social-distance violations from the same evaluation pass.
4. For ProcTHOR-style generation, save generator version and house seeds so the panel can be reconstructed.
5. For Habitat 3.0-style social tasks, record humanoid policy, interaction mode, and safety threshold.

Navigation and social benchmarks become useful when they preserve both route quality and interaction quality. A Habitat 3.0 social-navigation result should not report only whether the robot found and followed the person. It should also report whether the robot maintained safe distance, avoided blocking, and completed the task under the same humanoid behavior model as the baseline.

The graduate-level habit is to separate map competence from social competence. AI2-THOR and ProcTHOR can stress generalization across interactive indoor scenes and generated homes. Habitat 3.0 can stress collaboration with humanoid or human behavior. A paper-facing comparison should say which competence is measured and which scene or human-behavior distribution was held out.

A robust navigation evaluation starts with the episode panel. The panel should list each scene or generated house, start and target, shortest path length, seed, humanoid behavior if present, and the metric rule. Every method should consume that panel unchanged.

1. Write an episode panel with scene ID, generated-house seed, start, target, and shortest path length.
2. Freeze sensors, action space, success radius, collision rule, horizon, and social-distance threshold.
3. Evaluate every method through one script that computes success, efficiency, collisions, and social violations.
4. Save per-episode replays and stratify results by seen versus unseen scenes or generated houses.
5. For social tasks, report both task completion and human-aware safety metrics.

Code Fragment 2 records the navigation protocol fields that make a result replayable. The same schema works for a static navigation panel or a ProcTHOR-generated panel.

# Record a navigation result with path and social metrics together.
# Co-computing these fields prevents a method from optimizing route
# efficiency while hiding collisions or personal-space violations.
from dataclasses import dataclass, asdict

@dataclass
class NavigationResult:
    suite: str
    scene_split: str
    generated_seed_panel: str
    success_rate: float
    mean_spl: float
    social_violations_per_episode: float

    def as_row(self) -> dict[str, object]:
        return asdict(self)

result = NavigationResult(
    suite="Habitat 3.0",
    scene_split="unseen_homes",
    generated_seed_panel="proc_panel_v2",
    success_rate=0.74,
    mean_spl=0.51,
    social_violations_per_episode=0.18,
)
print(result.as_row())

Expected output: the printed result should expose scene split, generated-seed panel, success, path efficiency, and social violations. If one field changes between methods, the comparison should stay in diagnostics rather than the paper table.

When a navigation or social experiment fails, replay the path and tag the first failure mode: wrong target, inefficient route, collision, timeout, blocked humanoid, social-distance violation, object-state miss, or generated-scene mismatch. The tag tells you whether to fix mapping, planning, interaction policy, or the evaluation panel.