Calibration / docs / embedding

Refit the probabilities on your own outcomes. What temperature scaling and the conformal threshold each buy you, and when calibrating makes things worse.

The engine's published calibration is measured on the engine's own test split. Your traffic is not that split, and a probability that is honest there can be over- or under-confident here. Calibration fits a small correction per question on your own labelled examples, and then the probabilities, the prediction sets and the abstain flag hold on data like yours.

If you intend to threshold on a probability, or to gate on abstain, this page is not optional.

Two things are called calibrate#

This bit the author of these docs, so it goes first rather than in a footnote.

	vegaml.calibrate(...)	vegaml.calibrate.calibrate(...)
what it is	a function on the package	a function in the module of the same name
takes	a loaded model or a checkpoint directory	a loaded model
returns	a report dict	a Calibration object
effect	writes the calibration into the checkpoint directory, in place	attaches to nothing until you call cal.apply(result) yourself

Both are real and both are useful. The failure mode is assuming the second behaves like the first: a pipeline that fits, calibrates and then serves is serving raw probabilities while believing they are calibrated.

Watch out

The package spends one import binding to keep the name.

vegaml also ships a legacy vegaml/calibrate.py module, and the first import of any submodule binds it onto the package over whatever held the name. In 0.12.3 one fit() call could turn vegaml.calibrate into a module, and the next calibrate call died with 'module' object is not callable. Fixed in 0.12.4 by doing that binding at import time. If you see that error, upgrade.

The persistent one: refit a checkpoint#

This is the shortest path to vega-U on your own business, and it does not require attaching anything:

python

import json, vegaml

cal = json.load(open("my_labels.json"))     # the shape below

v = vegaml.load()
report = vegaml.calibrate(v, cal["examples"], cal["questions"])
print(report["ece_before"], "->", report["ece_after"])

out = v.decide("The dishwasher shows error E-114 and I already restarted it twice.",
               cal["questions"])
print(out["answers"]["category"]["choice"], out["answers"]["urgency"]["score"])

The refit lands on the loaded model immediately and is written into your local copy of the checkpoint, so later vegaml.load() calls on this machine keep it. The Hub copy is never touched: a fresh download restores the shipped calibration.

What it covers: the temperature (physics-conditioned where that beats a plain one on a holdout), the conformal prediction-set thresholds, the abstain thresholds and the unbound threshold.

The file format

questions maps each id to its definition, and every example is [state, {question_id: label}], which is the same shape Vega.fit takes.

json

{
 "questions": {
  "complaint": {"type": "boolean", "instructions": "Does the text describe a problem?"},
  "category":  {"type": "choice",  "instructions": "Which area does this text belong to?",
                "criteria": {"billing": "money, invoices, charges or subscriptions",
                             "technical": "devices, software or technical faults",
                             "operations": "logistics, facilities, incidents or travel",
                             "other": "anything else"}},
  "urgency":   {"type": "score",   "instructions": "How urgent is the text?",
                "criteria": ["no urgency at all", "slightly time-sensitive",
                             "moderately urgent", "clearly urgent",
                             "critical, immediate action needed"]}
 },
 "examples": [
  ["I was charged 49.99 twice this weekend. Refund the duplicate or I cancel.",
   {"complaint": true, "category": "billing", "urgency": 3}]
 ]
}

The labels are just your historical outcomes, which is why this is the place your own business plugs in: a support desk labels which team took the ticket, an insurer which claims were escalated, a lender which applications defaulted. The ids, the categories and the scale are yours to define, and the engine answers those questions with probabilities calibrated on those outcomes.

The explicit one: a Calibration object#

Use this when you want the calibration as a file you control, applied in a serving process, rather than written into a checkpoint directory.

python

from vegaml.calibrate import calibrate, Calibration

examples = [("I was billed twice, refund me or I cancel.", {"churn": "true"}),
            ("The export button does nothing on Safari.",  {"churn": "false"}),
            # ... a few dozen
           ]

cal = calibrate(v, examples, questions, alpha=0.1)
print(cal.report)                   # per question: temperature, ECE before and after

out = cal.apply(v.decide(state, questions, mode="engine"))
cal.save("calibration.json")        # Calibration.load() in the serving process

That apply call is also what gives a fitted head an abstain bit in the first place, by attaching the conformal prediction set and the flag the raw head never had. So the two traps on that page are really one trap: calibrate, and apply it, or your model has no way to say it does not know.

What it fits, and what it never does#

Per question, two numbers, in order:

A temperature on the probabilities, minimising negative log likelihood on your labels. Above one it softens overconfidence, below one it sharpens timidity.
A conformal threshold on the corrected probabilities, so the prediction sets cover your labels at 1 - alpha on data exchangeable with the examples. Where the model is wrong often, the sets widen. That honesty is the feature.

apply() moves probabilities, confidence, prediction_set and abstain. It never changes which option wins: the argmax is invariant under temperature. If the answers are wrong, calibration cannot help; fit a head on the same labels instead. The two compose, and that is the usual end state: a head for the answer, a calibration for the probability.

How many labels#

calibrate() refuses a question with fewer than min_examples (default 10) and leaves it exactly as the engine answered it, saying so in the report. Thirty to a hundred per question is where the temperature stops moving; 500 is where the library's own guidance says the numbers are usable.

One caveat on the report: its ECE figures are measured on the fitting examples themselves. Hold rows out if you want an unbiased number, which is what the three-way split on the fitting page is for.

When calibrating makes it worse#

We measured this properly on a two-way guard across four corpora, with calibration fitted on its own split and ECE reported on a test split neither fit had touched. Temperature scaling made calibration error worse on three of the four, and the one improvement came from a run where the temperature railed to the floor of its own search range, which is what a near-separable split looks like rather than a good fit.

There is also a structural limit worth knowing before you expect an abstain bit from a two-option question. vegaml marks an answer abstain when the conformal prediction set is empty, and the set keeps every label clearing 1 - qhat. With two options the winner always has probability at least 0.5, so abstention requires qhat below 0.5. On two of our four corpora qhat came out at 0.68 and 0.82, and the guard could not have abstained on any input at all.

Note

So the recommendation is narrow.

Calibrate if you want the abstain bit and your qhat lands under 0.5, and route what it abstains on to a second, slower check. Do not calibrate expecting more trustworthy probabilities on a two-way question; on that task it made them less trustworthy three times out of four. The full numbers, including the per-corpus temperatures and quantiles, are in the guard benchmark write-up.

Measure it on your own data rather than taking either claim on faith: vegaml-eval reports ECE and coverage on held-out rows, which is the comparison that decides it.