Section 23.5: Data quality, diversity, and labeling

"Every dataset has a junk drawer. The question is whether you labeled it."

A Patient Data Curator

Quality As A Labeling Problem

An episode has at least three truth layers: what the operator intended, what the robot executed, and what the environment actually did. Labels should distinguish these layers. "Failure" is too coarse; "object slipped after contact", "operator intervened", "camera dropped frames", and "reset distribution mismatch" teach different lessons.

A useful data-quality score can be written as a weighted checklist:

$$Q(e) = w_s S(e) + w_c C(e) + w_t T(e) + w_l L(e) - w_i I(e),$$

where $S$ is task success, $C$ calibration validity, $T$ timing validity, $L$ label completeness, and $I$ intervention burden. The weights should be stated in the dataset card rather than rediscovered from code.

The following example scores episodes and routes them into splits. Notice that the rule keeps stress data rather than deleting every imperfect row.

# Route episodes by quality gates instead of using one vague keep flag.
# Stress and repair examples stay useful because their failure type is explicit.
episodes = [
    {"id": "a", "success": 1, "calibrated": 1, "synced": 1, "labels": 1, "interventions": 0},
    {"id": "b", "success": 0, "calibrated": 1, "synced": 1, "labels": 1, "interventions": 2},
    {"id": "c", "success": 1, "calibrated": 0, "synced": 0, "labels": 0, "interventions": 0},
]

for e in episodes:
    score = e["success"] + e["calibrated"] + e["synced"] + e["labels"] - 0.5 * e["interventions"]
    route = "train" if score >= 3.5 else "stress" if e["labels"] else "repair"
    print(e["id"], score, route)

The expected output sends episode a to training, episode b to stress evaluation, and episode c to repair. That routing is the mechanism behind useful curation: the pipeline does not ask whether an episode is good or bad in the abstract. It asks what scientific role the episode can play after calibration, synchronization, labels, and interventions are known.

Annotation Schema

Mechanism: Separating Outcome From Cause

Outcome labels answer whether the task succeeded. Cause labels answer why it succeeded or failed. A robot can fail because the perception system localized the object incorrectly, because the gripper command saturated, because the operator intervened late, or because the reset placed the object outside the intended distribution. These cases should not be merged into one negative class because they imply different repairs.

For training, cause labels can support recovery data, filtering, or curriculum design. For evaluation, they allow per-failure reporting: a new policy may improve missed grasps while leaving occlusion failures unchanged. That is a more useful research result than a single aggregate score with no diagnosis.