Section 12.6: Reading a leaderboard without fooling yourself

A Careful Control Loop

What This Section Builds

This section makes leaderboard reading operational. It gives you a checklist for deciding whether a result compares methods, or whether it accidentally compares different tasks, splits, seeds, simulators, wrappers, or tuning budgets.

The goal is not cynicism. The goal is disciplined trust: promote only construct-matched metrics that were co-computed in one pass on one configuration, and keep everything else in diagnostics.

Theory

A leaderboard row is a summary of many design choices. The visible number may be success rate, SPL, reward, normalized score, completion rate, or throughput, but the hidden denominator is the episode panel and evaluation protocol. If two rows use different denominators, subtracting them does not estimate a method effect.

The practical design rule is simple: table comparisons need one evaluation script, one config, and one artifact that contains all compared rows. Cross-paper numbers can motivate discussion, but they should not be written as direct wins unless the benchmark protocol is demonstrably the same.

Worked Example

Code Fragment 1 checks whether rows can share a paper table. It rejects a direct comparison when one method uses a different split, seed set, or metric, even if the row names come from the same benchmark family.

# Group leaderboard rows by the protocol fields that make them comparable.
# A method win is paper-facing only when all compared rows share
# panel, split, seed set, wrappers, simulator, and metric.
rows = [
    {"method": "baseline", "panel": "MT50", "split": "official", "seeds": (0, 1, 2), "metric": "success"},
    {"method": "candidate", "panel": "MT50", "split": "official", "seeds": (0, 1, 2), "metric": "success"},
    {"method": "ablation", "panel": "MT50", "split": "tuned_validation", "seeds": (0, 1, 2), "metric": "success"},
]

groups = {}
for row in rows:
    key = (row["panel"], row["split"], row["seeds"], row["metric"])
    groups.setdefault(key, []).append(row["method"])

print(groups)

Practical Recipe

1. Identify the construct: manipulation generalization, lifelong transfer, long-horizon household progress, navigation efficiency, social safety, or simulation throughput.
2. Check whether every compared row shares the same panel, split, seeds, wrappers, simulator build, and metric script.
3. Prefer seed-level values and confidence intervals over one best-run number.
4. Audit whether validation data, public test episodes, prompt templates, or generated-scene seeds influenced tuning.
5. Keep mismatched cross-paper numbers in a diagnostic note rather than a win table.

Reading a leaderboard well means converting a ranked list into an evidence map. A ManiSkill3 speed result, a Meta-World success result, a Habitat SPL result, and an Isaac Lab throughput result can all be valuable, but they answer different questions. The audit is to keep each number attached to the question it actually answers.

The graduate-level habit is to distinguish result validity from comparison validity. A row can be valid for its own protocol and still invalid as a direct comparison to another row. A paper-facing claim needs both: the row must be internally reproducible, and the compared rows must be co-computed on one configuration.

A robust leaderboard entry starts as a machine-readable row with provenance fields. Store the model checkpoint, benchmark version, split, seeds, wrappers, simulator, metric script, hardware when throughput is measured, and the raw per-episode outputs. A reviewer should be able to reconstruct the table from that row.

1. Build a provenance table before copying any result into prose.
2. Group rows by task panel, split, seeds, wrappers, simulator, and metric.
3. Promote only groups with two or more methods to direct comparison.
4. Attach uncertainty or seed-level values to every promoted comparison.
5. Write mismatched rows as diagnostic context, not as wins.

Code Fragment 2 shows a row format that supports direct-comparison filtering. It is deliberately boring, because boring provenance is what keeps benchmark claims honest.

# Store leaderboard provenance with the metric value.
# The comparison key is everything except method and value because
# those are the fields a method is allowed to change.
from dataclasses import dataclass, asdict

@dataclass
class LeaderboardRow:
    method: str
    panel: str
    split: str
    seeds: tuple[int, ...]
    simulator: str
    metric: str
    value: float

    def as_row(self) -> dict[str, object]:
        return asdict(self)

row = LeaderboardRow(
    method="candidate",
    panel="Habitat-SocialNav",
    split="unseen_scenes",
    seeds=(0, 1, 2, 3, 4),
    simulator="Habitat 3.0",
    metric="success_weighted_by_path_length",
    value=0.51,
)
print(row.as_row())

Expected output: the printed row should expose the provenance fields needed to reconstruct the comparison. If a leaderboard omits these fields, treat it as scouting evidence until the configuration can be verified.

When a leaderboard comparison looks surprising, audit the denominator before interpreting the method. Check for hidden split changes, seed tuning, wrapper drift, simulator drift, metric changes, hardware changes for throughput, and selective reporting. Only then decide whether the method result needs a scientific explanation.