One CSV of cases your team already decided: a column of records, a column per question, the answer a human gave. That single file does two jobs, fitting and calibrating, and they fail separately and cost different amounts of data.
Before the upload: it already answers#
The API works out of the box, before even a project exists: POST /v1/decide takes the questions in the request. With a project but no upload, the engine answers from your written criteria alone, with probabilities through the checkpoint's published calibration. Every answer says readout: "engine" so you can always tell a cold answer from a fitted one, and the response shape never changes, so the integration you write on day zero is the one you keep.
What day zero does not give you is a number measured on your own traffic. The cold path is how you check the sheet reads your records the way you meant it; everything below is how the answers and the numbers become yours.
The split comes first#
The file is shuffled, then cut 60 / 20 / 20. Shuffled because case-system exports arrive in date order, and cutting that in sequence hands the last slice a different month.
| slice | share | what it does |
|---|---|---|
| fit | 60% | trains a small classifier on top of the frozen reader |
| calibration | 20% | measures the temperature and the conformal threshold |
| held out | 20% | touched by neither; every number you are shown comes from here |
Every number you are shown comes from the third slice, because a model always looks good on rows it learned from.
Fitting changes the answers#
The reader stays frozen. Nothing is fine tuned, no gradient reaches the language model, and your rows never enter a model anybody else uses. What gets trained is a small classifier over pooled features: standardise, reduce to 128 dimensions, then a multinomial logistic regression.
That is what makes your labels mean something: your rule about which refund disputes go to billing lives only in your examples, and fitting moves the answers toward it.
Note
A head that loses is not served.
A fitted head has to beat the frozen engine on cross-validated rows before it is allowed to answer anything. One that does not is refused rather than deployed, and the screen says so in those words. On our own corpora that takes a few hundred labelled rows per question, so the usual outcome of a small upload is that nothing switches over.
Calibrating changes what the numbers mean#
Calibration does not touch the answers. It touches what the confidence attached to them is worth, and it is measurement rather than learning.
A temperature is fitted per question, because a raw model runs hot: it says 0.95 when it is right about 80% of the time. Across six corpora this took the calibration error from 0.160 to 0.069, measured on held-out rows. That number is the average gap between stated confidence and observed accuracy.
A conformal threshold is then read off the same slice. It is not fitted at all; it is a quantile of a sorted list. Coverage and abstain works it through on real numbers.
How many rows you actually need#
What changes above the minimum is not accuracy so much as which parts of the answer exist at all.
| rows in the file | what the upload buys |
|---|---|
| under 50 | refused outright |
| 50 | a temperature per question. The shortlist is built but can never come back empty, because a threshold measured on ten rows sits below every answer |
| about 100 | the shortlist starts working: it narrows to one answer where the evidence is there and empties where it is not |
| about 200 | a 1 in 20 miss rate becomes available instead of only 1 in 10, and the review band has enough automatic rows for its precision to be a measurement |
| 300 and up | the fitted head starts beating the frozen engine, which is the only reason to fit rather than only calibrate |
Watch out
Those are counts per question, not per file.
Calibration only counts rows carrying an answer for the question it is calibrating, so five questions on one sheet do not share a pool. A column your team filled in on half the rows can be left uncalibrated while the rest pass, and the fit report names which ones and why.
Relabelling means remeasuring#
Change your refund policy and the old calibration is stale, because it was read off cases judged under the old rules. The threshold is not a setting you carry forward. Refit, which re-splits, which remeasures.
# the same two steps without an account, on your own rows
vegaml-calibrate fit rows.jsonl --out calibration.json
vegaml-calibrate check answers.jsonl --calibration calibration.json