Zero-shot, the engine answers your sheet from the criteria alone. That is often enough, and on adversarial or unusual text it is often not: the answers drift toward a constant, which looks like a strong model until you measure both classes. A policy is what fitting encodes.
Nothing in the language model is trained. The reader stays frozen. What you fit is a small head over the pooled features it already produces, which is why a few hundred examples is enough and why it takes seconds on a laptop rather than a GPU hour.
The one-call version#
examples = [({"body": "cancel my account"}, {"churn": "true"}),
({"body": "how do I export?"}, {"churn": "false"}),
# ... 20 or so, and a few hundred is better
]
report = v.fit(examples, questions)
print(report["churn"]["cv_accuracy"]) # 5-fold, stratified
print(report["churn"]["used_by_auto"]) # whether decide() will actually use it
out = v.decide(state, questions) # mode="auto"
print(out["answers"]["churn"]["readout"]) # 'ttt' here, 'engine' for anything unfittedAn example is (state, {question_id: label}), or (state, labels, {question_id: question}) when the question itself differs per row. Labels may be partial: a row can label one question and say nothing about the others.
The label values are the question's own option names: the choice key, "true" or "false" for a boolean, the level index for a score. A label outside the option set is reported in the per-question report rather than raising from inside the fit.
fit throws away heads that learned nothing#
This is the part worth trusting the library on. fit cross-validates every head and discards any that does not beat chance, because such a head has learned nothing and its probabilities are noise wearing a sharper distribution. "auto" is stricter still: it serves a head only when its cross-validated accuracy beats both chance and the engine's own accuracy on the same examples, so a thin head cannot displace a strong calibrated engine.
A head that beats chance but loses to the engine stays available to a deliberate mode="ttt". Nothing is hidden from you; it just is not served by default.
| report field | means |
|---|---|
fitted | a head exists for this question |
cv_accuracy | stratified 5-fold accuracy on the examples you passed |
used_by_auto | mode="auto" will serve it, because it beat the engine too |
reason | why it was not fitted: too few examples, one label only, or a label outside the option set |
The low-level head, when you want the knobs#
v.fit is the convenient path. When you want to hold the features yourself, sweep the one knob that matters, or fit on a split you control, the head is two numpy calls:
import numpy as np, vegaml
from vegaml.ttt import TTTHead
v = vegaml.load()
# One pooled feature vector per request. The question is part of the input, so
# features for one sheet are not features for another.
X = np.asarray([v.features(t, QUESTION) for t in train_texts])
# PCA down to k components, then multinomial logistic on top. That is the head.
head = TTTHead(["allow", "block"], k=96, l2=0.01)
head.fit(X, train_labels)
# Honest before you believe it: 5-fold on the training rows, never on the test rows.
print("cv %.3f" % head.cross_val_accuracy(X, train_labels))
p = head.predict(v.features(new_text, QUESTION), allowed=["allow", "block"])
pick = max(p, key=p.get)| argument | default | what it does |
|---|---|---|
k | 128 | PCA components kept. The knob that matters. |
l2 | 0.01 | ridge penalty on the logistic fit |
folds | 5 | cross-validation folds, stratified by label |
Sweep k before you draw a conclusion. Too low and the head underfits. On one of our sheets, moving k alone took a reader from 81.5% to 90.7%. A disappointing first number is worth a sweep before it is worth a verdict.
How many labels#
The library's own guidance is roughly 500 labelled examples per question type to be usable, and 1,000 to 2,000 before the numbers settle. Six is the hard floor below which fit refuses.
Be honest with yourself about what a small fit demonstrates. Our guard was fitted on 150 rows and should be read as a demonstration of the method rather than a tuned policy, and we say so on that page too.
Watch out
Never fit and test on the same rows.
It sounds obvious and it is the most common mistake in this area, including in our own first run. Split your labels three ways: 40% to fit the head, 20% to fit the calibration, 40% to test. Calibration fitted on the rows the head trained on is describing memorised data, so the probabilities look better than they are and the conformal quantile comes out too tight, which means the guard abstains less often than it should.
from sklearn.model_selection import train_test_split # or do it by hand
fit_rows, rest = train_test_split(rows, train_size=0.4, stratify=y, random_state=7)
cal_rows, test_rows = train_test_split(rest, train_size=1/3, stratify=y_rest, random_state=7)Two traps that cost us a day#
A head-served answer has no abstain bit
A fitted head returns probabilities, a confidence and a choice, and that is all. There is no abstain key and no prediction_set. Code doing answer["abstain"] keeps working right up until a fit succeeds and the head starts serving, which is the worst possible moment to find out.
The fix is calibration: cal.apply(result) is what attaches conformal prediction sets and the abstain flag to an answer that never had them. Until then, use answer.get("abstain", False) and know that you are not gating on anything.
Calibration does not attach itself
Two different things share the name calibrate in this library, and they behave differently. The Calibration page spells both out; the short version is that the module-level one returns an object attached to nothing, and a pipeline that fits, calibrates and then serves is serving raw head probabilities while believing they are calibrated.
Heads are plain numpy arrays with no save method of their own, so persisting one is your job. Calibration.save and .load are provided.