What the physics engine is worth

Where this sits, as of today

Every other system in this class is one model, fine-tuned or post-trained. The decision capability lives inside that model's weights, so changing which model reads for you means shipping a different checkpoint. vega-U splits the two: a 14.3M engine in a shared space, and a small linear port per reader.

Two things that table should not be read as claiming. The mechanism is not new research: aligning one frozen model's representation space to another is model stitching, with prior art from relative representations, Procrustes latent translation and latent functional maps. What is unusual here is the shape: the decider is deliberately tiny and frozen, so the reader is the only part that changes, and the alignment is label-free and runs as one call on text you bring.

And the rows other than ours come from their own documentation and third-party write-ups rather than from runs of our own, so treat the descriptions as a map of the field rather than as a benchmark. The only systems measured anywhere on this page are vega-U, laya and Jev.

Four things about this model are worth measuring rather than asserting, and all four fit on one laptop in an afternoon.

1. Ablate the physics engine against a linear softmax head with everything else held fixed, and say how much difference the engine actually makes.
2. Run all three systems cold, zero-shot, with nothing tuned, on corpora none of them were fitted on.
3. Find out what a calibration is worth once the data moves, how many labels it takes to put it back, and how often that has to happen.
4. Say who this is for, and why they would run a model themselves rather than rent one.

The short version, before the tables: we lose the cold comparison on all six corpora, clearly. The physics engine is worth between 8 and 63 labelled rows against a plain linear readout of the identical features, which is a much smaller number than our own marketing implies. With a hundred to a couple of hundred of your own labels a local head matches or beats the hosted model on three of six, and on reading comprehension nothing we run gets close. Fitted and calibrated, which is how vega-U would actually ship, it passes the hosted model on one corpus and closes most of the gap on two more.

On the hosted model's own kind of ground, zero-shot, our largest reader is 1.75 points behind it and ahead of the other open model, on a board whose own sampling error is wider than that gap. It is ahead outright on the ordinal score rail, which is what a risk score or a severity band is. And it has never been fitted, which is the measurement that moved the 0.8B from 0.530 to 0.679.

Where else we win is mostly the probability rather than the answer, and there is more of it than we first wrote down. Some configuration of vega-U is better calibrated than the hosted model on all six corpora, by 6.5x on the one where our accuracy is worst. Our conformal guarantee came out at exactly its 90% target after the corpus changed underneath it, while the hosted model's fell under target at 87.0%. We are faster on all six and the run cost nothing on our side. And averaging two small readers' probabilities, which needs no new training, is level with or ahead of the hosted model on three of six. Those are set out after the losses.

What was run

Six corpora, 2,155 rows in all, none of them on vega-U's published benchmark board. Every corpus is posed as the same kind of question, a choice with one line of criteria per option, so one scoring path covers all of them and the hosted model is asked the identical thing. Classes are balanced where the corpus allowed it, because accuracy on a corpus that is mostly one class rewards answering that class always, and one of the systems here does exactly that on two of these corpora.

One fixed, stratified test split per corpus carries every number below, so the zero-shot rows and every fitted head are scored on identical cases. The rest of each corpus is the pool the heads draw their labels from. Seed 7, inputs truncated to 1,200 characters before any system sees them, the hosted model pinned to jev-1.14.0 rather than jev-latest so a re-run scores the same model.

Hardware: one Apple M5, 16 GB, macOS 26.5.2. One process at a time, two BLAS threads. The Jev calls went one at a time over the public API from this machine and cost $0.0373 in total, with 0 failed requests.

A correction we owe you first

While setting this up we found that every local latency figure we have published, including the ones in the guard benchmark and on the benchmarks page, was described as measured on CPU. It was not. vegaml.load() with no device argument prefers CUDA, then Apple's Metal, then CPU, and on this machine that means Metal. Same cases, same sheet, one process at a time:

CPU is 5.9x slower on short text and 3.5x on long passages. So “runs on CPU” is true, and “119 ms on CPU” was not. The honest figure for a server with no GPU is about 704 ms a decision on this class of machine. We have corrected the other pages rather than quietly leaving them.

One: what the physics engine is worth

The ablation holds everything fixed except the readout. The same frozen Qwen3.5-0.8B reader produces the same pooled feature vector for the same (state, question) pair, and then either

• the engine settles a particle in that space and reads the answer off the wells, with no labels at all, or
• a linear softmax head is fitted on those vectors from labelled rows: standardize, then multinomial logistic. It reuses vegaml's own fitting routine, so the optimiser, the L2 and the iteration count are the shipped ones and the only thing removed is the physics.
• or that same head with the PCA step the shipped TTT head uses, 96 components, to show the projection is not what is doing the work.

Because the engine needs no labels and the head does, the comparison has to be a curve rather than a pair of numbers. The question is not “which is better”, it is how many labelled rows the engine is worth. Five seeds per point, the subsample stratified, the test split fixed.

So, stated plainly: on five of six corpora the physics engine is worth between 8 and 63 labelled rows against a linear readout of the identical features. That is the honest size of it. It is not nothing, and it is not the order of magnitude a reader would infer from the way we usually describe the engine.

The sixth corpus is the interesting one. On boolq no head reached the engine's accuracy at any label count we could afford, so there the settling computation is extracting something a linear probe on the same vectors does not get to. Both are poor in absolute terms, 0.575 against a 0.550 ceiling for the heads on a two-option question, so read it as a direction rather than a win.

What the engine gives that a head does not

Accuracy is the wrong column to settle this on, and this is the part of the ablation we would actually defend. A linear head has a confidence in the sense that a softmax has a maximum. It has no calibration, no conformal set and no abstain bit. Compare the calibration error of the two readouts at the point where their accuracy is comparable:

A head fitted on a dozen rows is confidently wrong: its calibration error sits between 0.288 and 0.522 at the low end of the curve, against an engine that is between 0.020 and 0.137 with no labels at all. On the moderation corpus the engine is the best-calibrated thing in this entire post at an ECE of 0.020, while being the least accurate at 0.421. It is accurately reporting that it does not know, which is a real property and a poor headline.

That is the trade the engine actually makes, and it is worth stating in one line: the engine buys you a usable probability with zero labels, not accuracy. If you have a few hundred labels and do not need the probability, a linear head on the same features is cheaper, faster and often better, and it is in the package as TTTHead.

Two: cold, out of the box

Nothing fitted, nothing tuned, the same sheet and the same truncated strings for all three. This is the table we would have preferred not to publish.

The reader here is the 0.8B, with gemma-1b beside it, because those are the two that run on the machine this was measured on. The 27B is further down, on vega-U's own board, which is the only place a reader that size appears.

Jev wins the accuracy column on all six. laya is second on all six. Our zero-shot engine is last on all six, and on two of them it is worse than guessing the majority class: 0.421 on moderation and 0.437 on counterfactual, where the majority class alone would score 33% and 78%. The “most-picked option” column shows why: on those two it answers one option 76% and 79% of the time. A guard against exactly this is the degenerate() check we shipped for the guard work, and it would fire here.

The gap on boolq is the largest and the most instructive: 0.919 against 0.575. That is a reading-comprehension task over a passage, and the reader doing the reading for us there is a 0.8B. Our own per-source board says the same thing in the other direction: long-context accuracy goes from 0.831 to 0.941 between the 0.8B and the 27B reader. The answer to boolq is a bigger reader, which costs a GPU, not a better engine.

Two things do go our way, and both are about the probability rather than the answer. Jev's calibration error on the two social corpora is 0.181 and 0.187, against 0.099 and 0.110 for the engine. That is not a fluke of our setup: the independent RLCDAlignBench study reports a median per-benchmark ECE of 0.168 for Jev against a null of 0.074, with 24 of 31 benchmarks above the null's 95th percentile, and attributes it to a base-rate mismatch rather than to bad ranking. Our 0.18 lands on their 0.168. Their framing is the useful one: Jev's probabilities rank extremely well, which is what its AURC column here shows, and they do not transfer as thresholds.

And on speed, the local models win by the length of the network. laya answered in 16 to 84 ms and the vega-U engine in 114 to 326 ms, both on this laptop's GPU, against 297 to 380 ms for the hosted call. Note that the hosted figure is a third measurement and a third answer: the same pinned model answered at a p50 of 775 ms in our guard run and 412 ms in the one before it. Accuracy reproduces across those runs and latency does not, so a single run of hosted latency should not be quoted as a specification, including by us.

How many of your own labels it takes to catch the API

This is the number that decides whether anyone would run a model themselves rather than rent one. The curve above, read against the hosted row:

On counterfactual a linear head on frozen features passes the hosted model at 64 labelled rows. On finance-sentiment and tweet-sentiment it lands within about a point with a couple of hundred, and both curves are still rising at the cap. On moderation it closes about half the gap, and on boolq it closes none of it.

We are not going to round that up into a claim. What it supports is a narrow statement: for a short-text classification decision where you have a few hundred labelled examples of your own, a model on your own hardware is competitive with the hosted one, and the remaining gap is worth less than the deployment properties to some buyers and worth more to others. For reading comprehension over documents with a small reader, it is not competitive at all.

The 27B against the hosted model, on vega-U's own board

The corpora above are the ones this post chose, and the reader on them is the 0.8B because that is what fits on a laptop. vega-U also publishes its own zero-shot board, four sources it never trained on, and that board is where the larger readers appear. It is the only ground on which a 27B reader and the hosted model can be compared, so it carries the headline comparison.

Three of its four sources rebuild verbatim from the training notebook's own converters: the same public dataset and split, the same answer type, the same instruction wording and the same criteria. That gives 1,200 items on which laya and the hosted model could be run for the first time.

1.75 points separate the hosted model and our largest reader, zero-shot, on ground the hosted model has no particular advantage on. On the corpora earlier in this post, with the 0.8B doing the reading, the same comparison was 11.5 points. The 27B is also 6.0 points clear of the other open model.

By source rather than overall, which is the more useful read. The 27B takes the score rail outright, 0.497 against 0.398 on a five-level ordinal question, with the 4B close behind at 0.492: reader size buys the ordinal task, and the ordinal task is what a risk score or a severity band actually is. The hosted model's remaining lead is almost entirely prompt injection, 0.855 against 0.730, the one source here that is a security judgement rather than a linguistic one. Emotion is within three points.

What changes once there is customer data

Everything above is zero-shot or a bare linear probe. Neither is the product. What vega-U ships as its strong configuration is a head fitted on the customer's labels and a calibration fitted on labels the head never saw, so that is the row that belongs beside the hosted model.

The split is 40/20/40 over each corpus: 40% fits the head, 20% fits the temperature and the conformal quantile, and the remaining 40% is the same test third every other row on this page is scored on. Fitting the head and the calibration on the same rows is the trap the middle slice exists to avoid. The temperature, the quantile and the abstain bit come from vegaml's own calibration code rather than a reimplementation, so what is measured is the shipped path.

One reconciliation before the numbers, because two tables in this post now report a fitted accuracy and they disagree. The ablation curve's last point gives the head the whole training pool, 240 rows on a 400-row corpus. The fitted row gives the head two thirds of that pool and spends the other third on the calibration. So the same labelled budget buys either a slightly more accurate head or a slightly less accurate head with a probability you can threshold, and on tweet-sentiment that is 0.686 against 0.660. That is the trade, and it is a real one rather than a rounding artefact.

Against the zero-shot engine, fitting is transformative on five of six. On counterfactual it reaches 0.883 and passes the hosted model. On finance-sentiment it goes from 0.547 zero-shot to 0.761, and on moderation from 0.421 to 0.629. The hosted model is still ahead on four of six.

boolq is where fitting fails outright. The head lands at 0.500, which is below the zero-shot engine's 0.575, and its cross-validated accuracy on the training rows is 0.481, under the 0.5 chance line for a two-option question. A head that does not beat chance has learned nothing, and Vega.fit discards exactly that case rather than serving it, so in production this row would not exist: the engine would answer. We are printing it because we ran it.

Fitting erases the difference between readers

vega-U ships several readers of the same engine, so the same fit was run on a second one, Gemma 3 1B, with the identical split, head and calibration recipe. Zero-shot the two readers disagree by as much as 32 points on a single corpus. Fitted, they land within 0.7 of a point of each other on the mean.

Gemma is ahead on tweet-sentiment and finance-sentiment and its cross-validated accuracy on the training rows beats the 0.8B's on three of six. That is worth more than a reader comparison: it says the zero-shot gap between readers was mostly the readout rather than the features. The pooled features of a 1B carry nearly as much task signal as the 0.8B's, and replacing the zero-shot readout with a head trained on labels recovers almost all of it.

What the calibration bought

Calibration helped here, which is the opposite of what we found on the guard task. Calibration error improved on five of six, and on two of them by a lot: moderation from 0.167 to 0.047 and boolq from 0.359 to 0.045. It got worse on tweet-sentiment-shifted. Accuracy is identical in both rows by construction, because temperature scaling is monotonic and cannot change which option wins; what moves is the probability attached to it.

Two caveats that belong next to that table. boolq's temperature railed at 20.0, the ceiling of the search range, which is the optimiser failing to find an optimum on a head that learned nothing rather than a fit. And abstention never fired on any corpus. An answer abstains when the conformal set comes out empty, the set keeps every label clearing 1 - qhat, and the winner always holds at least 1/k of the mass, so an empty set needs qhat below 0.667 on a three-option question and below 0.500 on a two-option one. Every qhat here came out above its threshold. The escalation band that the whole design argues for is, on these six corpora, not reachable at alpha 0.1.

Three: what a calibration survives, and how often to refit

Calibration was fitted on a held-out slice of each corpus, a temperature and a split conformal quantile at alpha 0.1, and then evaluated three ways: on an i.i.d. slice, on a slice of the same corpus with the class mix deliberately skewed to about 80% one class, and on a different corpus asking the identical question with the identical options.

The finding is clean and it is about which kind of drift matters.

A change in the class mix breaks calibration, every time. On all five corpora with a prior-shift case the calibration error got worse, and on two of them it roughly doubled or worse: moderation from 0.033 to 0.308 and boolq from 0.056 to 0.269. Conformal coverage mostly survives, because the quantile is deliberately conservative, but the probability stops meaning what it says. This is the same failure the RLCDAlignBench authors diagnose in the hosted model, which suggests it is a property of the problem rather than of either vendor.

A change in the text, with the same question, is survivable. The covariate-shift case moves from one Twitter sentiment corpus to a different one collected separately. The calibration fitted on the first held on the second: ECE 0.116 raw became 0.055 calibrated, and conformal coverage came out at 90.0% against a 90% target. That is the case we were least confident about and it is the one that held.

How many rows to put it back

Refitting on rows from the new distribution, holding the rest out:

Sixteen rows already lands the calibration error under 0.08, and it stops moving by about sixty-four. The cost of the sets tells the rest of the story: at 16 rows the conformal quantile is coarse and over-covers at 98.9% by keeping wider sets, and by 64 it is close to the target with tighter ones.

So, on how often to refit, with the hedges attached:

• Trigger on the class mix, not the calendar. The thing that broke calibration in this experiment was the mix of answers, and that is observable without any labels: you already see the distribution of the model's own answers. A sustained move in it is the signal to recalibrate, and it is cheap to watch.
• Watch the unbound rate too. It is the separate flag for inputs that landed off the calibrated map, and it is the covariate-shift alarm the ECE will not raise on its own.
• Budget tens of labels, not thousands. Sixty-four recent labelled rows per question restored calibration here. That is a weekly or monthly sample from a review queue you already staff, not a labelling project.
• A quarterly floor underneath both, because a trigger you have never tested is not a control. The published guidance in this area is vendor blogs rather than evidence, and the one peer-reviewed thing we found, anytime-valid calibration monitoring, exists precisely because periodic schedules and threshold alarms do not control the false-alarm rate over continuous monitoring. Treat a fixed cadence as a backstop.

And the field-engineering share of that, stated plainly because it is a commercial question: the first calibration is ours, because it needs the question sheet, the split discipline and someone who has seen a railed temperature before. Two of the six fits in the table above railed at a search bound, which a customer would not recognise. The recurring recalibration should not be ours. It is one call on a labelled file, and if a customer still needs us for it in month three, we built the wrong thing.

What of this is field-engineering work, and what is not

Stated plainly, because it is a commercial question and the honest answer is not “all of it”. The first calibration is ours. It needs the question sheet written, the three-way split held, and someone who recognises a railed temperature: two of the six fits in the table above hit a search bound, which a customer would read as a number rather than as a failure.

The recurring recalibration should not be ours. It is one call on a labelled file, it needs sixty-four rows, and the trigger is watchable without labels at all. If a customer still needs us for it in month three, we built the wrong thing. What stays with us after month one is the part that is genuinely engineering: a new question sheet when the business adds a decision, a reader swap when the hardware changes, and reading the eval when a number moves.

What it adds up to

The losses are above and they are the larger part of this run. These are the places the same measurements put us ahead, every one of them read off the same test splits and the same cases as the tables that go against us.

Calibration, on all six corpora

Some vega-U configuration is better calibrated than the hosted model on every corpus here, and the fitted-and-calibrated gemma reader beats it on all six by itself.

The moderation row is the one worth quoting, because it is the row where our accuracy is worst. The zero-shot engine is 6.5x better calibrated there at an ECE of 0.020. A model that is 42% accurate and reports that honestly can sit behind an escalation band. One that is 76% accurate and overstates its confidence is harder to put a threshold on.

Accuracy, on the corpus that rewards reading the criteria

Three configurations pass the hosted model on accuracy here, and the fitted row also takes AURC, 0.036 against 0.042, which means its confidence ordering ranks better as well as its answers being right more often. This is the corpus whose label is a linguistic property rather than a topic, which is exactly what a written criterion can describe.

Two small readers beat either one alone

We hold pooled features from two readers over the identical cases, so three combinations cost nothing more to test: fit on each alone, fit on the two concatenated, or average the two single-reader heads' probabilities. Averaging the decisions is the one that works.

The probability average reaches a mean of 0.691 and is level with or ahead of the hosted model on three of six: a tie on tweet-sentiment at 0.692, ahead on counterfactual, and 0.6 of a point behind on finance-sentiment. Concatenating the features does almost nothing, which is the useful detail: the gain comes from two readers making uncorrelated errors, not from having more columns to fit on. Two 1B-class readers on a laptop is a cheaper route to that than one large one.

The guarantee survives the corpus changing, and the hosted one does not

Given the same recipe, the same 120 rows to fit it on and the same corpus to spend it on, whose probability survives? Jev exposes a distribution per answer, so the identical temperature-and-conformal fit runs on it unchanged and no system gets a recipe the others did not.

At alpha 0.1 the point of the conformal set is that it holds the true answer 90% of the time. After the corpus changed, the vega-U engine came out at 90.0%, exactly on target, and the hosted model at 87.0%, under it, having also been under it at home at 89.3%. Its sets are the tightest at 1.68 options against our 2.02, which is worth having only if they hold.

Faster on every corpus, at no per-request cost

Between 1.1x and 2.8x on the 0.8B and up to 4.0x on gemma, and the whole hosted side of this run cost $0.0373 against nothing on ours. The honest reading is that this is a property of where the model sits rather than of the silicon: ours is compute with no network, theirs is mostly a packet in flight.

The one corpus where the physics is irreplaceable

On boolq a linear head on the engine's own features never reached the engine's accuracy at any label count the corpus could supply, topping out at 0.550 against 0.575. On the other five the engine is worth 8 to 63 labelled rows and is then overtaken. There it is not overtaken at all.

And one that is in no table because the other systems do not expose it: an answer here carries abstain, unbound and a conformal prediction_set as fields, fitted on a held-out split, which is what lets a gateway route a middle band rather than threshold a number somebody guessed.

Four: who this is for, and why not the hosted one

The honest answer is narrower than it could be. Our first customer is a regulated financial institution in a jurisdiction with a data-residency rule: a bank, an insurer, an NBFC, a payment operator, a broker. Not because the model is better at finance, but because that is the one buyer for whom the deployment property is the product, and for whom the alternative is not a cheaper vendor but no vendor at all.

Everywhere else, the cold comparison above is the one that matters, and on the live-traffic corpus in our guard benchmark the hosted model beat us on every axis while being given no labels at all. We are not going to pretend otherwise to win an argument about where the weights live.

What actually rules a hosted API out, and what does not

The published descriptions of Jev agree on three facts that decide this: it is cloud only, it is served from the United States, and there is no self-hosted or on-premise edition. Third-party reviews put retention as “as long as necessary” per the privacy policy, with zero retention available to enterprise accounts on request, and say customer data is not used for training. Those are secondary sources, so a buyer should confirm the retention terms and the DPA with the vendor rather than with us or with a blog.

None of that is a criticism of the model. It is a statement about where bytes go, and for most workloads it is a non-issue. It becomes the whole issue in a short list of places:

Two caveats on those rows that we are not going to bury. The dates are genuinely contested in the secondary sources: on DPDP, most compliance write-ups put full compliance at 13 May 2027 while several say the penalty provisions switch on from 13 November 2026, and we could not reconcile them, so check the Gazette notification. And the RBI published a draft Guidance on Regulatory Principles for Model Risk Management in June 2026 covering AI and machine learning models, which we found in only one source and could not confirm against the RBI's own site. Anyone making a deployment decision should read the primary text and take advice. We are a model vendor, not your counsel.

Why a frozen reader plus a 14.3M engine is the shape that fits

Four properties, none of which is an accuracy claim:

• It runs inside the perimeter. The default reader is 1.77 GB. On the laptop used for this post it decides in about 120 ms on the machine's own GPU and about 700 ms on CPU alone, measured above. There is no egress to register, no cross-border transfer to paper, and no vendor uptime in front of a payment queue.
• The weights are public and pinnable. A model risk function can name a version, hold a checksum, and re-run last quarter's decisions against this quarter's file. Behind an API whose version alias moves, that exercise is not available. TypeSafe's own guidance to pin a versioned identifier rather than jev-latest exists because the alias does move.
• The probability is refitted on the institution's own outcomes, and the refit is a file they hold. That is the mechanism behind a challengeable decision: an answer carries a number, that number was fitted on this bank's labelled history, and the threshold that routed the case to a human is written down.
• Nothing is generated. There is no prose to hallucinate, so the audit question is about a distribution over a fixed option set rather than about what the model said. For a regulated decision that is a much smaller surface to defend.

And one property that is not ours and should not be sold as if it were: the 27B reader that scored best on our board is a 27B model, and running it in a bank means buying the GPU to run it in a bank. The 0.8B that runs on a laptop is the one that makes the deployment argument, and it is also the weakest reader we publish.

The ₹1.64 crore email, and what would have stopped it

On 24 October 2026 Kochi City Police registered an FIR on a complaint from the Lulu Group's legal manager. Press reporting of that FIR describes a loss of about ₹1.64 crore, put at €150,500, to a business email compromise. The scheme ran from 2 to 29 September. Addresses resembling those of IE Park SRL, a long-standing Italian partner, carried invoices and new bank details for an outstanding balance on gaming equipment, and the money went to the accounts the impostors supplied. The case is under the Bharatiya Nyaya Sanhita and the IT Act, no suspects had been identified at the time of reporting, and the allegations have not been tested in court.

We are using it because it is the ordinary shape of the loss, not the exotic one. The FBI's IC3 counted 24,768 BEC complaints and $3.05 billion in reported losses in 2025, the second-costliest category it tracks, and its Recovery Asset Team froze more than $507 million in fraudulent domestic wires, which is the measure of how much of this turns on catching it in hours.

The controls that stop it are process, not models

This needs saying before anything about what we sell. The control that would have stopped this payment is a callback, and the detail of the callback is the whole control:

• Call a number from the vendor master file, a signed form or the vendor's official site. Never the number in the email, and never the signature block, because in vendor email compromise the attacker is often inside the supplier's real inbox and changed the contact details first.
• Make the vendor read the account out. Do not read them the numbers from the email and ask whether they are right, which lets an attacker confirm their own figures.
• Dual authorisation above a threshold, where the second approver reads the callback evidence rather than the same email the first approver read.
• Hold the first payment after any change of bank details for manual review.

A decision model does not verify a bank account and cannot replace any line of that list. What it can do is cheap and specific: read every inbound message and answer “does this ask to change where money goes?” so that the callback workflow is triggered by the message rather than by whoever happened to open it. That is a two-way typed decision with a probability on it, which is the one thing that makes it routable: high confidence holds the payment, the middle band goes to the payments team, low confidence does nothing.

PAYMENT_CHANGE = {
    "type": "choice",
    "instructions": (
        "This message arrived in an accounts-payable inbox. Decide whether acting "
        "on it would change where money is sent. Judge the request, not the tone: "
        "a polite, well-formatted message from a known supplier is the usual shape "
        "of this request."
    ),
    "criteria": {
        "changes-payment": (
            "asks for payment to a different account, bank, country or intermediary "
            "than the one on file, supplies new banking details, attaches an invoice "
            "whose remittance details differ from the supplier's existing ones, or "
            "asks for an existing payment to be redirected, held or reissued"
        ),
        "ordinary": (
            "anything else: a quote, a statement, a query, a dispute, a reminder "
            "about an unpaid invoice that does not change the destination"
        ),
    },
}

We have not benchmarked that sheet, and we are not going to quote a number for it. There is no public corpus of real business email compromise that postdates these models; the spam corpora that exist are on our own training board, so a figure from them would be meaningless; and a corpus of BEC emails we wrote ourselves would measure our imagination rather than the model. The sheet above is the shape of the thing, offered as a starting point for someone who has their own mail archive and their own labels, which is the only data that can answer it. That is also the honest version of what a field engagement does: the labels are the customer's, and so is the measurement.

The rest of the BFSI surface

Every row below is the same primitive: a state, a typed question, a calibrated probability, and an explicit band that goes to a human. None of them needs generated text, which is why none of them needs a frontier model, and all of them involve records that a residency rule or a liability rule touches.

What is measured on this page and what is not matters here. The six corpora above include one BFSI-domain task, financial headline sentiment, and nothing else in that table has been benchmarked by us. The rows are a map of where the primitive fits, not a claim about accuracy on any of them. The number that should decide a deployment is the one from the institution's own labels, which is what vegaml-eval exists to produce.

What this run told us to fix, and what the fix measured

Three gaps in everything above are ours rather than the benchmark's, and all three fall out of these numbers rather than out of anybody's opinion. They are closed in vegaml 0.14.0. What follows is what the fixes measured, on the same laptop, because a fix announced is not a fix.

The escalation band had no way to fire

The whole case for putting this inside a bank is that the uncertain middle goes to a human. So it matters a great deal that in the fitted runs above, the abstain flag fired on zero rows out of 860, and could not have fired on any of them.

The reason is a definition. Once you recalibrate on your own labels, which is the step this post spends a third of its length recommending, abstaining means the conformal prediction set came out empty. For that to be possible at all, the conformal threshold has to sit below 1 minus one over the number of options. It did not, on any corpus:

What ships instead reads confidence and never looks at the prediction set. You can set the line yourself, per question, with no labels involved. Or, with labels, you can ask for the thing you actually want, which is not a threshold but a standard: give it the precision the automatic band has to hold and it returns the widest band that holds it, along with what that costs in review load.

v.set_band({"payment_change": 0.8})          # a line you choose
out["answers"]["payment_change"]["route"]   # "auto" or "review"

cal = calibrate(v, examples, questions, target_precision=0.97)
cal.report["payment_change"]["band"]
# {"threshold": 0.83, "review_rate": 0.31, "precision": 0.983,
#  "precision_lb": 0.971, "held_out": {"precision": 0.975, ...}}

Two details decide whether that number survives contact with a pilot. The target is held on a one-sided 95% lower bound rather than on the precision the band happened to score, because the threshold is chosen on the same rows that then grade it, and picking the best-looking cut on a few hundred rows is exactly how a 97% target ships as a 91% deployment: twelve correct out of twelve certifies 0.816, not 1.0. And a target that nothing can hold is refused rather than approximated. Every answer routes to review, and the report names the target this much data could actually support. The failure mode we were careful to rule out is the one where the band quietly gets weaker than the number written on the contract.

The CPU number was mostly configuration

703.8 ms per decision on CPU against 118.7 ms on Metal is the figure published above, and a bank's on-prem box often has no GPU at all. Most of that gap turned out to be a setting. The right thread count belongs to the reader, not to the engine, and torch's default is wrong for the one we ship:

The shipped reader gets 4.9x slower going from one thread to four and 14.0x slower going to ten. It is not the engine and not the prompt length: Qwen3.5's linear-attention blocks fall back to a reference PyTorch implementation whenever the CUDA kernels are missing, which on a CPU they always are, and that fallback does not thread. A plain-transformer reader has no such fallback and behaves the ordinary way. It holds at length too, 547 against 1639 ms at 855 characters, and 1019 against 2853 ms at 3426 characters.

The thread count is free in the only sense that matters: one thread and four agreed on 12 of 12 choices with a confidence difference of 0.0000. The second lever is the reader, and it is free in a different sense: the engine behind every reader is the same 14.3M parameters, checksum-identical, so this is a configuration change rather than a different model. Fitted and calibrated across the six corpora, the gemma reader means 0.672 accuracy against the 0.8B's 0.679, at a calibration error of 0.059 against 0.069. Seven tenths of a point behind on accuracy, a point ahead on calibration, and 135.2 ms instead of 703.8 ms. That is 5.2x, with no quantization and no GPU in the box.

Recalibration is one command, and it tells you when to run it

If a customer still needs us in month three we built the wrong thing, which is easy to write and worth checking. So recalibration is one command, and so is the question of whether it is due:

vegaml-calibrate fit --labels recent.jsonl --questions @questions.json \
    --target-precision 0.97 --out calibration.json

vegaml-calibrate check --answers last-week.jsonl --calibration calibration.json
# exits 1 when the traffic has moved off the calibration

The check needs no labels at all, which is the point: labels are what the refit costs, and this is the thing that tells you to go and collect them. It reads answers the deployment has already produced and compares the mix of them against the mix the calibration was fitted on, which the fit records automatically. It exits non-zero at the point of action, so a cron entry and a pager are the entire alerting setup.

What it watches is decided by the drift section above rather than by taste. A covariate shift, the same decisions written differently, left calibration intact and in fact improved it. A change in the mix of answers broke it on every corpus we tried. So the statistic is the population stability index over the answer mix, which is one counter per option and is already what model-risk teams in banking read, at its conventional bands: under 0.1 stable, 0.1 to 0.25 watch, 0.25 and over refit. Beside it sit mean confidence, because a reader that has started hedging on everything moves no mix at all, and review load, because a band whose review rate has doubled is costing a pilot money whatever the statistics say. An option the calibration never produced is not drift in degree but a different question, and always a refit. Under fifty answers for a question it reports the numbers and refuses a verdict, because a mix drawn from a handful of answers moves on its own, and paging on that is how a drift alert gets switched off for good.

The bill for acting on it is the one already in the short version: 26 seconds on 240 rows, almost all of it reading each row once, with the temperature and the conformal quantile together costing 6 ms. Sixteen labelled rows restored calibration after the prior shift and it stopped moving by sixty-four. That is a morning of labelling, not an engagement.

What this does not measure

• One reader. Everything local here is the Qwen3.5-0.8B reader, the smallest we publish, because it is the one that runs on a laptop. The 4B and 27B readers score materially higher on our own board and were not run here.
• Heads, not the fitted product. The curve fits a head on frozen features. It is not the same thing as vega-U's own fitted-and-calibrated path, and the numbers should not be read as that path's ceiling.
• Contamination is not controlled. boolq, amazon_counterfactual and tweet_eval are widely published and predate all three systems, so all three may have seen them. That inflates every zero-shot row roughly equally and cannot explain a degenerate answer distribution, so the findings against us are conservative under it.
• 240 labels is a cap, not a plateau. Several curves are still rising where they stop, because a 400-row corpus cannot supply more while keeping a 160-row test split.
• Latency is one laptop. Treat it as a ratio between rows. The hosted figures carry the network from one machine in India to a US region on one afternoon.
• No BEC benchmark. As set out above, there is no public corpus that would make one meaningful.

Run it yourself

The harness is one file. It caches every stage, so each can be re-run without paying for the ones before it, and the corpora download to the ordinary HuggingFace cache rather than being copied into a repository.

pip install vegaml pandas pyarrow huggingface_hub
export JEV_API_KEY=...          # only needed for the hosted rows

python -I scripts/ablation_eval.py \
    --n 400 --n-twin 300 \
    --curve 8,16,32,64,128,240 --curve-seeds 5 \
    --stages features,engine,laya,jev,ablate,drift

Swap the task table at the top of the file for your own corpora and the same three measurements run against your data, which is the only version of this that should decide anything.

Sources

• Corpora: cardiffnlp/tweet_eval, mteb/tweet_sentiment_extraction, zeroshot/twitter-financial-news-sentiment, tdavidson/hate_speech_offensive, google/boolq, mteb/amazon_counterfactual.
• Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures, for the independent zero-shot and calibration results on the hosted model.
• Red Hat Developers, benchmarking AI decision models against traditional guardrails, an independent guardrail comparison putting Jev 1.13 at 86.35% and laya at 85.44% on prompt injection.
• O'Reilly Radar on Jev, for the spam comparison: 98.3% zero-shot against 98.4% for a TF-IDF model trained on about 14,800 labelled emails.
• FBI IC3 2025 annual report, for the business email compromise totals.
• Reporting of the Kochi FIR: DT Next, Metro Vaartha, The420.
• Google Cloud on RBI payment data localisation, and the IRDAI Maintenance of Information regulations, 2025.
• RBI's FREE-AI committee report, August 2025.
• Our own harness and this run's raw output are in the vegaml ecosystem repositories, and the guard study it follows is here.

If one of these numbers is the one that matters to you, the harness takes your corpus in about ten lines of diff. We would rather you ran it than believed us.