Safety engineering for learned drivers

Beyond a good model

The previous safety module asked when a learned model should not be trusted. This one asks the engineering question around it: what do you put around a learned driver so that its failures do not become the vehicle's failures, and how do you show that it works? Three pieces answer it.

1. A safety envelope: simple, checkable rules that can override the learned policy.
2. Evidence: how much driving it takes to claim a failure rate, and what that does to the design.
3. A safety case: the argument that ties the pieces together.

The same shape appears in Deploying a driving VLA, where a fallback exists for the moment a deadline is missed: the learned model is the performer, and something simpler is the guard.

A rule that can overrule the network

Responsibility-Sensitive Safety (RSS, 1708.06374) is a formal model of what a careful driver does. Its longitudinal rule gives the smallest gap from which a following car can always stop in time, under worst-case assumptions about the car in front. With rear speed $v_r$, front speed $v_f$, a reaction time $\rho$, the most the rear car may accelerate during that time $a_{\max}$, the braking the rear car is guaranteed to use $b_{\min}$ and the hardest the front car may brake $b_{\max}$:

Read it as three distances. The rear car travels $v_r\rho + \tfrac12 a_{\max}\rho^2$ while it reacts, then needs $(v_r + \rho a_{\max})^2 / 2b_{\min}$ to stop from the speed it has reached; the front car needs $v_f^2/2b_{\max}$ to stop, which it uses up before the rear car arrives. If the gap is at least $d_{\min}$ the rear car cannot hit the front one, whatever the front car does within the assumptions. If it is smaller, the rule says to brake at $b_{\min}$ until it is not.

def rss_min_gap(
    rear_speed: torch.Tensor,
    front_speed: torch.Tensor,
    reaction: float = 0.5,
    accel_max: float = 2.0,
    brake_min: float = 4.0,
    brake_max: float = 8.0,
) -> torch.Tensor:
    """The smallest gap (m) from which a rear car can always stop in time.

    Worst case assumed by RSS for the same lane: the rear car keeps accelerating at
    ``accel_max`` during its ``reaction`` time, then brakes at only ``brake_min``,
    while the front car brakes as hard as ``brake_max``. If the gap is at least this
    large, the rear car does not hit the front car.
    """
    after_reaction = rear_speed + reaction * accel_max
    rear_distance = (
        rear_speed * reaction
        + 0.5 * accel_max * reaction**2
        + after_reaction**2 / (2 * brake_min)
    )
    front_distance = front_speed**2 / (2 * brake_max)
    return (rear_distance - front_distance).clamp(min=0.0)


def rss_filter(
    proposed: torch.Tensor,
    gap: torch.Tensor,
    speed: torch.Tensor,
    lead: torch.Tensor,
    brake_min: float = 4.0,
    reaction: float = 0.5,
) -> tuple[torch.Tensor, torch.Tensor]:
    """Replaces the proposed acceleration with a brake where the gap is unsafe.

    Returns:
        The acceleration to apply and a mask of where the filter intervened.
    """
    unsafe = gap < rss_min_gap(speed, lead, reaction=reaction, brake_min=brake_min)
    return torch.where(unsafe, torch.full_like(proposed, -brake_min), proposed), unsafe

Wrapping a driver in the rule

The filter takes the policy's proposed acceleration, and replaces it with a brake wherever the current gap is below $d_{\min}$. It does not need to understand the policy; it needs only the gap and the two speeds. We run it on the car-following scenarios from the closed-loop module, 300 scenarios with 30 percent hard-braking leads, around two drivers.

The first two rows are the filter doing its job: a driver that would collide in more than a third of the scenarios does not collide once the filter is on. The last two rows are the price. The expert never collided to begin with, yet the filter overrides it in a quarter of all time steps, because the envelope's idea of a safe gap (computed for the worst case) is larger than the gap the expert chose (13.6 m on average). The vehicle drives more conservatively, with gaps 28 percent bigger.

This is the central design trade-off, and the parameter $b_{\min}$ is where it lives. A smaller value (assume the car brakes gently) makes the envelope cautious and intrusive; a larger value makes it permissive. The choice must reflect what the vehicle can actually do on the road you are on: in the rain the braking it can rely on is lower. Setting $b_{\min}$ above what the car can deliver makes the filter look better in a table and leaves the guarantee empty. In our toy the largest value, 6 m/s², equals the ego car's maximum braking, which is as far as it can go.

What the envelope does not do

• It trusts its inputs. The rule is only as good as the gap and speeds it is given. A missed detection (see state estimation) is a missing gap, and the rule cannot protect against an object it was never told about.
• It covers what its authors modelled. The longitudinal rule is one of several; lateral manoeuvres, occlusions and right of way each need their own. The guarantee is "within these assumptions".
• The toy's lead car brakes at 5 m/s², under RSS's assumed maximum of 8. The 0-collision result is a property of the toy, not a statement that the RSS guarantee was met.
• Stopping is not always safe. A car that brakes to a halt in a live lane can be hit from behind. Real envelopes consider the response, not only the rule.

Even so, the structure is valuable. The learned policy can be improved freely, and the part that carries the safety argument is small enough to read, test exhaustively and certify.

Evidence: how much driving is enough?

Suppose you want to show that the vehicle fails less than once per $10^8$ miles. If you drive $n$ miles and see no failure, how confident can you be? If the true rate were $r$ per mile, the probability of no failure in $n$ miles is $e^{-rn}$. Setting that to 5 percent (95 percent confidence) gives

the "rule of three". For $r = 10^{-8}$ that is about 300 million failure-free miles. For $10^{-7}$, 30 million. The RAND report Driving to Safety (Kalra and Paddock, 2016) made this argument for autonomous vehicles; the arithmetic is the same here.

def failure_free_miles(rate_per_mile: float, confidence: float = 0.95) -> float:
    """Miles with no failure needed to claim the true rate is below ``rate_per_mile``.

    With zero failures in ``n`` miles, the chance of that happening at rate ``r`` is
    exp(-r n). Setting it to 1 - confidence gives n = -ln(1 - confidence) / r. At 95
    percent this is about 3 / r (the "rule of three").
    """
    return -math.log(1.0 - confidence) / rate_per_mile


def failure_rate_upper_bound(
    failures: int, miles: float, confidence: float = 0.95
) -> float:
    """Upper confidence bound on a failure rate after seeing ``failures`` in ``miles``.

    The largest rate r for which seeing this few failures (a Poisson count with mean
    r * miles) still has probability at least 1 - confidence. Found by bisection.
    """
    low, high = 0.0, (failures + 10.0 * math.sqrt(failures + 1) + 10.0) / miles
    for _ in range(100):
        mid = 0.5 * (low + high)
        mean = mid * miles
        cdf = sum(
            math.exp(-mean + k * math.log(mean) - math.lgamma(k + 1))
            for k in range(failures + 1)
        )
        if cdf > 1.0 - confidence:
            low = mid
        else:
            high = mid
    return 0.5 * (low + high)


def redundant_failure(p: float, common_cause_share: float, channels: int = 2) -> float:
    """Failure probability of ``channels`` redundant channels that all must fail.

    A share ``common_cause_share`` of each channel's failures is a common cause that
    takes all channels down together; the rest is independent per channel.
    """
    independent = ((1.0 - common_cause_share) * p) ** channels
    return common_cause_share * p + independent

Two consequences follow.

• Failures make it worse. If you see one failure in $10^8$ miles you can claim only that the rate is below about $4.7\times10^{-8}$ at 95 percent, not $10^{-8}$; the bound after seeing zero is $3.0\times10^{-8}$. Evidence has to be gathered, and every fix restarts the count for the changed part of the system.
• You cannot test your way to a very low rate. That is why evidence is a mix: simulation (the closed-loop module), proving properties of the small trusted parts (the envelope), and fleet data, with each doing what it can do cheaply. It is also the reason a small, simple guard around a big, learned model is attractive: you can argue about the guard directly instead of waiting for miles.

Redundancy, and why common cause limits it

Two independent channels, each failing with probability $p$, both fail with probability $p^2$. For $p = 10^{-3}$ that is $10^{-6}$. This is the appeal of redundancy, and it is the number you get if the failures are independent. They rarely are: a shared power supply, a shared software bug, the same weather degrading two sensors, the same training data biasing two models. A simple model splits each channel's failures into a share $\beta$ that is common to both and the remainder that is independent:

With $\beta = 0$ this is $10^{-6}$ again. With a 1 percent common cause it is $1.1\times10^{-5}$, eleven times worse; with 10 percent it is $1.0\times10^{-4}$, a hundred times worse. The common share dominates, so the engineering effort goes into independence: different sensors, different algorithms, different power, and a guard (like the envelope) that does not share the learned model's blind spots. Two networks trained on the same data are a weak form of redundancy; a network and a rule-based check are a stronger one.

How safety is argued

Engineering teams organise the evidence into a safety case: an explicit argument, backed by evidence, that the system is acceptably safe for its operating domain. Three standards frame the vocabulary.

• ISO 26262 covers functional safety: hazards caused by faults in electrical and electronic systems. It sorts hazards into automotive safety integrity levels (ASIL A to D) that decide how rigorous the development must be.
• ISO 21448 (SOTIF) covers hazards that arise even with no fault, because the intended function has limits: a perception model misses a rare object, a planner misjudges a situation. This is the standard that speaks to learned components.
• UL 4600 covers safety cases for autonomous products.

Faults in the hardware and software that run a learned driver still fall under ISO 26262. What a learned driver adds is SOTIF-style hazards: the work is to find the situations where the intended function is not good enough, which is the data-engine loop, and to show that the envelope and monitors cover what remains.

Why this matters for a VLA

A driving VLA is a performer without a guard. The Alpamayo-R1 repository states that the released model is not a complete driving stack and has no redundant safety mechanisms. Everything in this module sits outside the model: the envelope that overrides an unsafe trajectory, the monitors from the safety module, the fallback in the deployment module, and the evidence process. A team moving from a modular stack to a learned one should expect to keep, and rely on, much of the safety architecture it already has.

Reference implementation (PyTorch)

Tests are in libs/tutorial/tests/test_safety_engineering.py. They check the RSS gap against the formula by hand, that the filter overrides only where the gap is unsafe, that it removes the collisions of a driver that ignores the lead, that a more permissive $b_{\min}$ overrides a good driver less, the rule-of-three arithmetic, and that redundancy multiplies probabilities only when there is no common cause. Run uv run --group tutorial python -m libs.tutorial.safety_engineering.

Check yourself

1. What does the RSS longitudinal rule guarantee, and what does it assume?
2. Why does the filter override a good driver in a quarter of the steps, and what would you change to reduce that?
3. Why is setting $b_{\min}$ above the real braking capability a mistake even though it makes the numbers look better?
4. How many failure-free miles show a failure rate below $10^{-7}$ per mile at 95 percent confidence?
5. Why does a single failure change the claim you can make?
6. Two redundant channels each fail with probability $10^{-3}$, with a 10 percent common cause. Why is the result nowhere near $10^{-6}$?
7. What does SOTIF cover that functional safety does not?

Quiz

Answer each question; feedback appears as soon as you choose.

Sources

• Shalev-Shwartz, Shammah and Shashua, On a Formal Model of Safe and Scalable Self-driving Cars, 2017 (RSS).
• Kalra and Paddock, Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?, RAND Corporation, 2016.
• ISO 26262 (functional safety of road vehicles), ISO 21448 (SOTIF) and UL 4600 (safety cases for autonomous products), as standards documents; check the current editions before relying on a summary.
• Toy numbers (collisions, overrides, gaps, rule-of-three values) are from our reference implementation.