Once a model can decline to answer, a better question becomes available. Not “how accurate is it”, which is an average over cases you would never have automated anyway, but: what share of my traffic can I automate at a precision I choose?
You give the Route screen a target. It returns the widest automatic band whose precision holds at that target, and tells you what that band costs in human review. Both figures are measured on the held-out slice.
Why twelve correct out of twelve is 0.816#
The precision reported is a one-sided 95% lower bound, not the raw fraction. Twelve correct out of twelve certifies 0.816, not 1.0.
That is not pessimism, it is the honest reading of twelve observations. Twelve for twelve is perfectly consistent with a true rate near 82%, so 82% is what can be certified. A narrow band pays for being narrow, and a system that reports 1.0 off twelve rows is reporting a number that will not survive the thirteenth.
Note
Widening the band usually raises the certified precision.
Counter-intuitive until you see why: a wider band holds more rows, and more rows tighten the bound. Fifty correct out of fifty certifies far more than twelve out of twelve does, even though both are flawless.
When the target cannot be met#
Nothing is quietly downgraded. If no threshold in your data can hold the precision you asked for, the band does three things at once: it sets reachable to false, it puts the threshold above every observed confidence so that every answer escalates, and it names the best target your data could actually support.
{"reachable": false,
"best_achievable": 0.91,
"review_rate": 1.0}Lower the target, widen the review budget, or label more rows. Shipping a band that does not hold its own target is the one option not offered.
Budgeting the other way round#
You can also fix the review rate instead of the precision, which is the right shape for a pilot with a set number of reviewers. Give it the share of traffic you can afford to look at and it returns the line that sends exactly that much, with the precision reported rather than targeted.
| you fix | you learn | the shape of the question |
|---|---|---|
| a precision target | what share needs a person | quality first, staff to it |
| a review rate | what precision that buys | budget first, learn what it is worth |
What this is worth#
It converts a model into a staffing line: 500 cases a day at a band that automates four fifths means 100 land on a desk, and those are the ones where the model was shaky. Pick the target from what the two mistakes cost you: a wrong automatic refund costs the refund, a case sent to a person costs minutes.
The threshold itself comes from coverage and abstain, and it travels with your exported calibration.