Nobody's test split is your traffic, and the published numbers for any decision model, this one included, are measured on its own. vegaml-eval measures the model on your labelled decisions instead, which is the number that should decide whether you ship it.
vegaml-eval decisions.jsonl --questions @questions.jsonEach JSONL row is one labelled state:
{"state": "I was billed twice, refund me or I cancel.", "labels": {"churn": "true"}}--questions is the same sheet decide() takes, inline JSON or @file. A row may carry its own "questions" where the instruction varies per example. From Python it is one call:
from vegaml.evals import evaluate
report = evaluate(v, rows, questions)What the report says#
Per question, and pooled overall:
| number | what it measures |
|---|---|
accuracy | against your labels, nothing clever |
ece | the gap between claimed confidence and measured accuracy: the honesty number |
coverage | how often the true label was inside the conformal set. At alpha=0.1 it should sit near 0.9. |
mean_set_size | how wide the set had to be to achieve that coverage. A set that covers by always holding every option is measured here, not hidden. |
abstain_rate | how often the case was too hard to commit on |
unbound_rate | how often the input sat off the calibrated map, reported separately on purpose |
latency_ms_p50 / p95 | and n_tokens_per_decision, overall |
Read coverage and mean_set_size together, always. Coverage alone is trivially satisfiable by a model that keeps every option in every set, and that model has told you nothing.
What to do with a bad number#
The three numbers point at three different fixes, and mixing them up wastes a day.
| symptom | what it means | fix |
|---|---|---|
| low accuracy, honest ECE | the engine does not know your task | fit heads |
| decent accuracy, bad ECE | it knows, but it is miscalibrated on your traffic | recalibrate |
| coverage far from 1 - alpha | usually the same thing as the row above | recalibrate, then eval again on held-out rows |
| many unbounds | your traffic has drifted away from the calibrated distribution | recalibrate on recent rows |
| many abstains | the cases are genuinely hard for the engine | fit heads |
Thirty to a hundred labelled rows per question is enough to start trusting the direction of these numbers. The eval costs one decide() per row, so a thousand rows is a coffee.
Watch out
Report both error rates, never one.
On any two-class question, a single headline number can be perfect for the wrong reason. A model that blocks everything catches every attack; a model that forwards everything never refuses a customer. Both score well on one metric and are worthless. vegaml.guard.score exists precisely to make that hard to do by accident, and degenerate() names the failure in one line. See the guard recipe.