FirstRead / Learning from missed cases

Catching more risk also raised false alarms

Added questions and rules caught more risk, but also flagged more no-risk messages.

Risk messages caught (recall)

88.1% → 96.0%

+7.9 percentage points · higher is better

No-risk messages incorrectly flagged

1.5% → 3.5%

+2.0 percentage points · more false alarms

Up = more risk caught. Right = more false alarms.

  1. 1

    Starting policy

    Where we started: one cutoff per question, per level.

    88.1% caught · 1.5% false alarms

  2. 2

    Follow-up questions for borderline turns

    Miss addressed: Weak signals that add up, with no one answer high enough on its own.

    95.4% caught · 3.2% false alarms

  3. 3

    A question about preparation

    Miss addressed: Goodbye letters and gathering means reached CRISIS only by accident, or not at all.

    96.0% caught · 3.5% false alarms

BHH test: 519 risk and 1,231 no-risk messages; flagged at ELEVATED or above. Each saved run uses its commit’s policy. Three versions, not a threshold sweep. Results predate a later fix informed by a test miss.