Section 55.5: Failure recovery, security, maintenance

For Failure recovery, security, maintenance, deployment quality is measured by the command stream, safety monitor state, and replayable evidence behind each command.

A Careful Control Loop

Problem First

Deployment does not end after first success. Robots age, networks change, credentials expire, sensors drift, and adversarial inputs can target physical behavior.

The practical question is therefore specific: which observation arrives, which state estimate is trusted, which action is allowed, which monitor can interrupt it, and which artifact proves the claim afterward?

Theory

Failure recovery and security should be modeled as reachable states of the deployed system, not as prose commitments in a launch checklist. If a sensor drops, a battery browns out, or a signing key expires, the architecture must define what the robot does next and who may authorize recovery.

A simple recovery model is a guarded state machine with modes $\{\text{nominal}, \text{degraded}, \text{recovery}, \text{safe stop}\}$. Security adds trust predicates over software origin, credential validity, and command authority. Maintenance adds slow-changing state such as battery health, encoder drift, and calibration age.

Worked Example

A delivery robot facing a blocked wheel and an expired service certificate needs two different recovery paths. One is physical and may require stopping or replanning. The other is cyber-physical and may require rejecting remote commands while preserving a local safe-stop channel.

recoverable_failures = [
    {"event": "wheel_slip", "recovered": True},
    {"event": "camera_drop", "recovered": True},
    {"event": "expired_certificate", "recovered": False},
    {"event": "stuck_lift", "recovered": False},
]

rate = sum(x["recovered"] for x in recoverable_failures) / len(recoverable_failures)
report = {
    "section": "55.5",
    "recovery_success_rate": rate,
    "requires_secure_maintenance_window": ["expired_certificate"],
    "requires_operator_repair": ["stuck_lift"],
}
print(report)

{'section': '55.5', 'recovery_success_rate': 0.5, 'requires_secure_maintenance_window': ['expired_certificate'], 'requires_operator_repair': ['stuck_lift']}

The expected output should split faults by the type of authority required to resolve them. That distinction matters operationally because a robot should not improvise its way through a security fault the same way it handles a temporary perception drop.

Practical Recipe

1. Write the observation, action, monitor, metric, and artifact fields before selecting a model.
2. Run a deterministic smoke test and one named perturbation from the panel.
3. Log success, safety events, latency, energy or resource use, and recovery status in the same row group.
4. Compare only methods evaluated by the same script on the same panel and seed plan.
5. Attach a short postmortem to each failed rollout so the artifact remains useful after the plot is forgotten.

Failure recovery, security, maintenance becomes operational when the metric is tied to a runtime interface. The interface names the sensor stream, state estimate, action representation, timing budget, safety or robustness monitor, and deployment artifact.

The disciplined habit is to separate three claims. The conceptual claim explains why the method should help. The systems claim explains which interface it changes. The evidence claim records which measurement would convince a skeptical builder.

Cross-References

For Failure recovery, security, maintenance, connect benchmark design, sim-to-real transfer, uncertainty, and safety barriers through the deployment artifact that will be checked before release.

When resilience fails, classify the incident as runtime fault, cyber trust failure, hardware degradation, or procedure failure. Then inspect whether the system entered the correct mode and whether the artifact preserved enough evidence to improve the next maintenance cycle.