Every decision API on the market returns a confidence score, and not one of them will tell you what it means. You get 0.9 and you are left to decide, on your own, whether 0.9 is enough to send money out of the building. There is nothing to check that number against, because nothing was ever measured to produce it.
That gap is the whole reason Embedding Studio exists. It is open today at studio.embedding.in, with $10 of credit and no card.
The two bad options
Say you want to automate refund decisions. A model looks at the call and says refund, 0.9. You now choose between two bad options.
Automate everything and you eat every mistake the model makes, including the confident ones, and you will not find out which those were until a month of refunds has gone out. Put a person on everything and you have bought a very expensive autocomplete.
What you actually want is the third thing: a model that tells you which decisions it is fit to make. Not a number you then threshold by instinct, but a threshold somebody measured.
What fitting actually does
You upload one CSV of cases your team already decided. One column of records, one column per question, with the answer a human gave.
The file is cut 60 / 20 / 20. The first slice fits a small classifier on top of a frozen reader. The reader is not touched, nothing is fine tuned, and your rows never enter a model anybody else uses. The second slice calibrates. The third is held back and never fitted on, and every number you are shown comes off that third slice.
A fitted head has to beat the frozen engine on cross-validated rows before it is allowed to serve. One that loses is refused rather than shipped, and the screen says so in those words. On our own corpora that takes a few hundred labelled rows per question, so a fifty row upload will usually fit nothing. We would rather say that than let you deploy a head fitted on thirty rows.
What calibration actually does
Two things happen on the calibration slice, and neither is learning in the sense that fitting is learning.
A temperature is fitted per question, because a raw model runs hot: it says 0.95 when it is really right 80% of the time. Across six corpora this took the calibration error from 0.160 to 0.069, measured on rows nothing was fitted on. That is the gap between the confidence a model states and the accuracy it actually delivers, cut to under half.
A conformal threshold is then read off the same slice, and this is the part no hosted API sells you. It is not learned. It is counted.
The bar, with real numbers
Take a hundred refund calls your managers already handled. For each one, look at a single quantity: how much probability the model put on the answer the manager actually gave. Not the winner. The correct one.
Sort those hundred numbers. Decide how often you are willing to be wrong, say one time in ten. Cross off the worst ten and read off where the survivors begin. Say that is 0.55.
That is the bar, and it took no tuning, no judgement and no config file. It falls out of the sort. And the guarantee is arithmetic rather than hope: those ninety survivors all had the manager's answer at 0.55 or above, so if the bar is 0.55 the correct answer was on the shortlist for all ninety of them.
Now run it:
- A clear double charge scores 0.97. Clears the bar. That refund goes out on its own and nobody reads it.
- Somebody past the refund window arguing their delivery was late scores 0.52. Under the bar. The answer comes back empty, and a person picks it up.
The second row is the entire product. A normal API would have returned "refund, 0.52" in exactly the same confident voice as the first, your code would have read the top answer, and the money would have gone out. Here nothing was returned, so nothing could be acted on.
Abstain is a field, not a convention
When no option clears the threshold, the answer is empty. Not "low confidence", not a flag you have to remember to read. There is nothing in the field.
The difference matters more than it sounds. A low score depends on somebody writing if confidence < 0.6: escalate, and on that 0.6 being the right number, and on the next engineer not deleting it. An empty answer depends on nothing, because there is no answer to read.
With three or more options you can also get a shortlist: two options clear the bar and both come back, so a person picks between two rather than reading the case cold. Three shapes, three costs: one answer is free, a shortlist is cheap, an empty answer is full price.
The number worth watching is your abstain rate. If 18% of cases come back empty, that is a staffing line you can budget. If it drifts to 35% next month, something changed in your traffic, and that is a smoke alarm you do not get from a model that answers everything.
Why twelve out of twelve is 0.816
Once you can abstain, you can ask the better question: what share of traffic can be automated at a precision I choose? Give the Route screen a target and it returns the widest automatic band whose precision holds at that target on a one-sided 95% lower bound.
Which is why twelve correct out of twelve certifies 0.816 and not 1.0. Twelve for twelve is consistent with a true rate near 82%, so that is what gets certified. A narrow band pays for being narrow, and a number that says 1.0 off twelve observations is a number lying to you.
Ask for a precision nothing in your data can hold and it does not quietly downgrade. It sends everything to review, sets reachable to false, and names the best target your data can actually support.
What it refuses to do
Four things, stated here because finding them out later is worse.
It will not run on too few rows. Fifty is the floor for an upload. But a question needs ten labelled rows in the calibration slice before it gets a temperature at all, and nineteen before the conformal threshold stops saturating, which is the first point an empty answer is even possible. Since the slice is a fifth of the file, abstain does not really work until around a hundred rows per question.
It will not serve a head that lost. Covered above. The usual outcome of a small upload is that nothing switches over.
The guarantee assumes your traffic keeps looking like your calibration rows. This is the real failure mode in production, not a footnote. Your case mix shifts, a product launches, attackers change their phrasing, and the bar you measured in March is wrong in September. Our own drift runs showed a covariate shift leaving a calibration intact and a prior shift breaking it on all five corpora, which is why the thing we watch is the mix of answers and why it needs no labels to watch.
It will not tell you it is more accurate than a hosted model. We run those comparisons and we publish them either way. On one guard benchmark of ours a hosted competitor caught more attacks than our own fitted head did. Accuracy is a per-task, per-corpus question, and anyone who answers it in general is selling something.
What we will claim is the part that is structural rather than empirical, and checkable against anyone's published documentation: a hosted decision API gives you no coverage guarantee that you set, no way for the model to answer nothing, and no review rate known before you ship. Those are not benchmark results. They are features that either exist in an API or do not.
Integrating it
One CSV to set up. After that it is one POST with a key, and the response carries a route field: auto means act on it, review means a person looks at it. The integration is a branch, not a threshold you had to invent.
Every response also carries an audit block naming the engine checksum, the reader, the engine version and the calibration version that produced it, so an answer from six months ago can be traced to the exact configuration that gave it. Send the same idempotency_key again after a timeout and you get the first answer back rather than a second charge.
Decisions are $0.0010 each. The request and response reference is in the product's own documentation, which is where it stays current.
Where this came from
The method was published before anybody was selling it: a March 2025 paper on reinforcement learning over sequence representations, open weights, an open dataset, a package on PyPI. Then laya, open sourced under Apache 2.0 and now community owned, which is not part of anything sold here. Then vega, then vega-U, the universal adapter. The whole arc is on the story page.
Through all of it the question in the inbox was never about the model. It was always: how do I know I can trust this on my data. Embedding Studio is the answer we can charge for, and the engine underneath it is still open source, still Apache 2.0, and still runs on your own hardware if that is what you need.
Open Embedding Studio, or read how it works first.