FirstRead / Technical appendix A5

Staying honest

A benchmark is easy to fool by accident. These are the rules we kept to.

Test messages never shape what Jev is asked. An early draft's examples echoed test messages. We caught it and rewrote them before measuring.
Tune on one set, report on another. BHH validation to tune, test to report; one later fix looked at a test miss, and the recall here predates it. 37 conversations to tune on, 21 held out, all labelled before any run.
Every gain has a cost. Recall went from 88.1% to 96.0%, and false alarms from 1.5% to 3.5%.

Still open

UnmeasuredThe reply check has no eval of its own.
Its cutoffs aren't measured on labelled replies yet. Late in a recovered conversation, it still replaces some calm ones.
Held backLive conversations can over-escalate.
Demo scenarios whose live runs go above their labels are held back until that's fixed.
No fallbackWhen the gateway fails, FirstRead fails closed.
Safe, but the message goes unread. There's no fallback classifier yet.
Close rivalglm-5.3-flash comes close.
AUC 0.976 against Jev's 0.992. Against it, Jev's lead is mostly speed: 0.50s median against 1.56s, 3.1× as fast.