Section 48.9: Closed-Loop Driving Evaluation And Safety Assurance

Open-loop scores measure how well you predict the world; closed-loop scores measure how the world changes once you act in it, and only the second kind keeps you safe.

On evaluating a driving stack

Open-Loop Metrics Are Necessary, Not Sufficient

Modern driving stacks are usually evaluated in at least three modes. Open-loop evaluation checks perception or prediction against recorded data. Closed-loop simulation checks how the full stack behaves when its own decisions perturb the future. Safety assurance asks a different question again: which hazards remain, what mitigations exist, and what evidence shows that the mitigations actually work inside the stated operational design domain?

The first two modes are often confused. A forecasting model can improve average displacement error while still making the planner too confident in the wrong interaction pattern. A smoother trajectory generator can reduce jerk while cutting safety margins. A benchmark score can rise even while the vehicle becomes easier to deadlock or easier to surprise in rare conditions.

Driving Assurance Mathematics

Closed-loop evaluation often reduces the stack to a score, but the useful object is a tuple of behavior and risk:

$$E_{\mathrm{drive}} = (c_{\mathrm{route}}, p_{\mathrm{infractions}}, m_{\mathrm{comfort}}, m_{\mathrm{interaction}}, d_{\mathrm{ODD}}).$$

Here $c_{\mathrm{route}}$ is route completion, $p_{\mathrm{infractions}}$ aggregates collisions and rule violations, $m_{\mathrm{comfort}}$ tracks jerk and acceleration, $m_{\mathrm{interaction}}$ tracks negotiation quality such as deadlock or unsafe gap choice, and $d_{\mathrm{ODD}}$ records how close the scenario lies to the edge of the validated operational design domain.

A useful benchmark formula is therefore not just "higher is better." One representative decomposition is:

$$J_{\mathrm{closed}} = c_{\mathrm{route}} \cdot p_{\mathrm{infractions}}, \qquad \text{subject to } d_{\mathrm{ODD}} \in \mathcal{D}_{\mathrm{validated}}.$$

This captures a core lesson from current leaderboard design: route progress matters, but it is only meaningful when infractions remain low and the scenario still sits inside the claimed ODD. Once the system leaves the validated domain, the benchmark score is no longer a deployment argument, only a research observation.

Current Toolchain And What It Is For

The practical tool stack for this section is: CARLA Leaderboard, nuPlan, Waymo Open Dataset, Waymo Sim Agents or Waymax, CommonRoad, Autoware, ASAM OpenSCENARIO, OpenDRIVE. Each tool answers a different assurance question. CARLA Leaderboard tests closed-loop route completion with explicit infraction metrics. nuPlan and CommonRoad structure planning evaluation. Waymo's public challenges and simulator tools expose interaction realism. Autoware makes stack interfaces concrete. OpenSCENARIO and OpenDRIVE keep the scenario and map assumptions portable.

Code And Evidence

We can put this into practice with a tiny assurance record. The goal is not to compress the whole safety case into one object. The goal is to force the stack to admit which metric improved, which infraction still happened, and whether the claim is actually promotable.

# Store one closed-loop assurance summary for a driving stack.
# Keep the route metric, infraction metric, and deployment decision together.
from dataclasses import dataclass, asdict

@dataclass
class AssuranceRecord:
    scenario_family: str
    route_completion_pct: float
    infraction_penalty: float
    blocked_events: int
    strongest_defeater: str
    decision: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = AssuranceRecord(
    scenario_family="unprotected_left_turn",
    route_completion_pct=96.0,
    infraction_penalty=0.91,
    blocked_events=1,
    strongest_defeater="late response to fast oncoming motorcycle in glare",
    decision="limit_to_daylight_until_glare_panel_passes",
)
print(record.as_row())

Expected output: the printed record should contain both performance and denial information, not just a success metric. If your assurance artifact cannot name the strongest remaining defeater and the resulting deployment limit, it is a benchmark summary rather than a safety argument.

Recipe For Builders

1. Write the ODD card before comparing models. A model that works in daylight urban traffic is not automatically a model for glare, heavy rain, rural roads, or emergency-vehicle interactions.
2. Use open-loop perception and prediction metrics as supporting evidence, then rerun the exact claim in closed-loop simulation.
3. Keep route completion, infractions, comfort, blocked-agent events, and intervention logs in one artifact.
4. Write the strongest defeater for every apparent success and convert it into a scenario or an explicit deployment limit.
5. Only promote the stack where the metrics and the written safety argument agree.