Troubleshooting

Every entry here is something that happened to us, in the order of how much time it cost.

The machine ran out of memory, or froze#

Almost always a batch. Batched prediction pads every member to the longest sequence in the batch, so one call with a few hundred long documents allocates a single attention tensor sized by the worst case times the count. It is not a slow run, it is an allocation that can take the desktop with it.

# wrong: one padded tensor sized by the longest of 762 jailbreak prompts
preds = agent.predict_batch(all_texts, questions)

# right
preds = []
for i in range(0, len(all_texts), 8):
    preds += agent.predict_batch(all_texts[i:i + 8], questions)

Three other things help, in order of effect: truncate the inputs to the length you actually intend to judge, cap the BLAS thread count (OMP_NUM_THREADS=2 MKL_NUM_THREADS=2), and call gc.collect() between models if you are scoring several in one process. With all four, our peak resident size on an eval over 2,000 cases was 0.8 GB.

401 from the Hub#

The Hub answers 401 both for “you are not allowed” and for “there is no such repository”, which makes a typo look like a permissions problem. vegaml's fetch turns that into one sentence naming which of the two it thinks it is and where it looked for a token.

• A gated reader (Gemma 3, for instance) needs HF_TOKEN set and the licence accepted on the Hub. The second one is easy to forget.
• Tokens belong to the reader, not to vega-U, so the published vega-U checkpoints load anonymously even when the reader you asked for does not.
• token=False ignores the environment and fetches anonymously, which is the fastest way to tell a bad token from a bad name.

'module' object is not callable#

On vegaml.calibrate(...), in versions before 0.12.4. The package also ships a legacy vegaml/calibrate.py module, and the first import of any submodule used to bind it onto the package over the function of the same name: one fit() call imported evals, which imports that module, and the next calibrate call died. Fixed by spending that one binding at import time. Upgrade.

KeyError: 'abstain'#

A fitted head returns probabilities, a confidence and a choice, and that is all: no abstain, no prediction_set. Code that reads those keys works right up until a fit succeeds and the head starts serving.

The fix is calibration: cal.apply(result) is what attaches the conformal set and the flag. Until then, answer.get("abstain", False) keeps the code running and tells you the truth, which is that you are not gating on anything.

ModuleNotFoundError on the image path#

decide_image needs vegaml[vision], which carries Pillow and torchvision; the backbone's processor loads torchvision on the way in and refuses to run without it. This raised in every installed copy before 0.8.0.

pip install "vegaml[vision]"

My accuracy is disappointing#

Work through these in order, because they are in order of payoff.

1. Read the criteria again. Options with no definitions, options that overlap, or no “other” option. Most of the quality lives here and it is prose, not code. See Questions and answers.
2. Sweep k. On one of our sheets, that knob alone moved a reader from 81.5% to 90.7%. A first number is not a conclusion.
3. Check whether the head is even being served. out["answers"][qid]["readout"] says ttt or engine. "auto" declines a head that loses to the engine, so a fit can succeed and change nothing.
4. Check whether the question varies per row. In intent classification the distinguishing text is often in the instruction, not the state. Pass the question per example or the head is fitted on identical prompts.
5. Try a bigger reader, last. It helps on reading tasks and does nothing on saturated short-text ones. The per-source board says which is which.

It says it is unsure much more often than I expected#

Many abstains

The cases are genuinely hard for the engine at your configured risk. Fit heads on labels for those questions, or lower the risk deliberately and accept the error rate that comes with it.

Many unbounds

Different problem: unbound means the input landed off the calibrated map, so your traffic does not look like the distribution the calibration describes. Recalibrate on recent rows. A steady trickle is normal; a rising rate is drift and worth an alert.

It never abstains on a two-option question#

Probably mathematics rather than a bug. An answer is marked abstain when the conformal prediction set is empty, and the set keeps every label clearing 1 - qhat. With two options the winner always has probability at least 0.5, so abstention is impossible unless qhat is below 0.5. Print it: cal.report carries it per question. On two of the four corpora in our guard run it came out at 0.68 and 0.82, and the guard could not have abstained on any input at all.

Loading is slow every time#

The reader is being re-downloaded. Check that ~/.cache/huggingface is writable and persistent: in a container that means a volume, as on the Docker page. A cold load is 27 s for the 0.8B and 61 s for the 4B; after that it is seconds.

If it is slow in a server rather than a script, start with --preload so the process does not bind until the model is resident, and have your health check read model_loaded rather than treating any 200 as ready.

An input was refused for being too long#

That is deliberate. An input that will not fit in max_len is refused rather than quietly shortened, because a model that trims your document and answers anyway gives you a confident answer about a document you did not send. Truncate on purpose, and then you know it happened. See Long context and images.