Benchmarks

Two measurements live on this page: the engine's own held-out board across readers, and the wall-clock speed of a decision. Both are stated with the split they came from, because a benchmark number without its split is a marketing number.

Four readers, one engine#

Trained jointly on Qwen3.5-0.8B and Gemma 3 1B (708,657 uid-matched items, records up to 73,728 tokens, with a cross-reader invariance loss pulling the same item's ported features together across tokenizers). The 4B and the 27B were attached cold afterwards, and every cold-attached reader scored above the readers the engine trained with, on an engine that is checksum-identical behind all four.

Held-out acc is with the published calibration on a split the engine never trained on. Zero-shot acc is the same engine on question sheets it never saw during training, which is the number that matters if your questions are your own. ECE is the gap between claimed confidence and measured accuracy. AURC and acc @ 50% cov describe what happens when you let it decline the hard half: lower AURC and higher accuracy at 50% coverage mean the confidence ordering is trustworthy, which is the property a cascade and an escalation band both depend on.

Where the gains land

This is the design's claim made measurable. From the 0.8B to the 27B, per source:

And where they do not. Saturated perception tasks stay flat across all four readers: Bitext intents 0.99, Enron-Spam 0.99, phishing 0.99, AG News 0.92 to 0.93. The ordinal score rail sits near 0.61 for every reader, which means that ceiling belongs to the engine or the labels, not to the reader. Paying for a bigger reader on a short-text classification task buys nothing.

Speed#

Per-decision figures from the T4 runs: 22 ms for the 0.8B reader and 28 ms for the 4B, engine and read together. That is a GPU with nothing else happening on it.

The table below is what a laptop actually gives you, measured end to end during the guard eval: an Apple M5, single process, two BLAS threads, one request at a time, over four corpora whose inputs differ in length. These are whole decisions including the read, not engine time.

The spread across corpora is input length, not variance: the shortest corpus is one line per case and the longest is 1,200 characters. Treat these as a ratio between rows rather than as a number for your hardware, and measure your own with out["latency_ms"], which every decision carries.

For context on what a network buys you, the hosted model in that same run answered at a p50 of 641 to 798 ms with a p99 of 1,123 to 1,687 ms and one request reaching 3,786 ms. A local decision and a hosted call are closer than the architectures suggest, because one is doing the work and the other is mostly waiting for a packet.

The guard benchmark#

Separate from the board above, we scored this engine as a prompt-injection guard against three published jailbreak corpora and one corpus of real user traffic, alongside a hosted decision model. It is the only benchmark on this site with a production base rate in it, and the result reversed the headline we expected.

• The recipe and the deployable numbers: Prompt-injection guard.
• The full study, every table, the contamination caveat and what we got wrong: Testing a decision model as a prompt-injection guard.

Measure it on your own data instead#

Nobody's test split is your traffic, this one included. The number that should decide whether you ship is the one from your own labels:

vegaml-eval decisions.jsonl --questions @questions.json

Thirty to a hundred labelled rows per question is enough to trust the direction. The Evals page explains what each number in the report means and which fix each bad number points at.

The harness behind the published board, the fine-tuning notebooks and the per-reader eval_report.json files are all in the vegaml repository, under Apache 2.0.