A confidence score on its own carries no promise. Conformal prediction replaces it with one that can be stated and checked: the correct answer is in what you get back about 9 times in 10, where the 9 is a number you choose rather than one we claim.
Where the threshold comes from#
Take the calibration slice, where you already know the right answer. For each row look at one quantity only: how much probability the model put on the answer the human actually gave. Not the winner. The correct one.
| the case | the human said | probability on that answer |
|---|---|---|
| charged twice, bank statement attached | refund | 0.96 |
| arrived smashed, photo attached | refund | 0.91 |
| used it three weeks, dislikes the colour | no refund | 0.88 |
| past the window, but delivery was late | refund | 0.42 |
| says it never arrived, tracking shows signed | no refund | 0.33 |
Sort all of them. Decide how often you will accept being wrong, say one time in ten. Cross off the worst tenth and read off where the survivors begin. Say that is 0.55. That is the threshold, and nobody tuned it.
The guarantee is then arithmetic rather than hope. Those survivors all had the human's answer at 0.55 or above, so with the bar at 0.55 the correct answer was in the set for every one of them.
Using it#
From then on the shortlist for any new record is every option scoring 0.55 or more. Three shapes come back, and they cost you different amounts.
| what cleared the bar | you get | what it costs |
|---|---|---|
| exactly one option | the answer | nothing, it is automated |
| two or more | a shortlist | cheap: a person picks between two |
| none | nothing at all | full price: a person reads it cold |
With a yes or no question the middle row cannot happen, because the bar sits above one half and two things cannot both exceed it. So a boolean is either answered or empty.
Abstain is a field, not a convention#
When nothing clears the bar, the answer is empty. Not a low number you have to remember to check. There is nothing in the field.
{"answer": null, "abstained": true, "route": "review"}The difference is larger than it looks. A low score depends on somebody writing if confidence < 0.6: escalate, on that 0.6 being right, and on the next engineer not deleting it. An empty answer depends on nothing, because there is no answer to act on.
Asking for more coverage widens the shortlist#
This goes the way round most people expect it not to. Demanding more coverage lowers the bar, because catching more cases means accepting weaker evidence, and a lower bar means more options clear it.
| you ask for | the bar | what happens |
|---|---|---|
| 8 in 10 | higher | nearly everything returns one answer, more mistakes get through |
| 9 in 10 | the default | the usual trade |
| 99 in 100 | lower | more shortlists and more empties, far fewer mistakes |
| 10 out of 10 | not available | see below |
Perfect coverage is not offered, for two reasons. It would make the output useless, because the bar drops to your single worst calibration row and then every option clears it on every record. And it is not available anyway: your threshold came from a finite sample, and a stranger case than any you have seen will eventually arrive. The quantile is capped at the size of the slice, which is the code being honest that the data cannot support the ask.
Watch out
The guarantee assumes your traffic keeps looking like your calibration rows.
This is the real failure mode, not a footnote. Your case mix shifts, a product launches, attackers change their phrasing, and a threshold measured in March is wrong in September. Our own drift runs showed a covariate shift leaving a calibration intact and a prior shift breaking it on all five corpora, which is why what gets watched is the mix of answers, and why watching it needs no labels. The early warning is your abstain rate climbing.
Row counts matter here more than anywhere else: the threshold does not become a real quantile until the calibration slice holds about nineteen rows for that question. See Fit and calibrate.