HTTP server / docs / embedding

vegaml-serve: POST /v1/decide returns exactly what decide() returns. Endpoints, errors, and how to scale it.

bash

pip install "vegaml[serve]"
vegaml-serve --model 4b --port 8000      # --preload to load before binding

Without --preload the checkpoint loads on the first request, so the server binds immediately and the first caller waits. With it, the process does not accept connections until the model is resident, which is what you want behind a health check.

Endpoints#

route	what it does
`POST /v1/decide`	{state, questions, mode?, alpha?, views?} in, the decide() dictionary out
`POST /v1/decide/batch`	{"items": [...]}, answered in order on the one loaded checkpoint
`GET /health`	{"status": "ok", "model_loaded": bool}
`GET /v1/model`	what is loaded: backbone, context limit, mode, fitted heads
`GET /docs`	the interactive schema

bash

curl -s localhost:8000/v1/decide -H 'content-type: application/json' -d '{
  "state": "I was billed twice, refund me or I cancel.",
  "questions": {"churn": {"type": "boolean",
                          "instructions": "Does the customer threaten to leave?"}}}'

The response is exactly what the Python API returns, so client code reads the same fields either way. Gate on abstain on this side of the wire too; see Questions and answers.

Errors#

A malformed request is a 422 naming the field. A bad question (unknown type, mode="ttt" without a fitted head) is a 400 carrying the model's own message. Both are worth logging with the request body, because both mean a caller is sending something it believes is valid.

Concurrency and scaling#

One process holds one checkpoint and serializes decisions around it. A decision is a single forward pass measured in tens of milliseconds, so the lock costs less than making per-call state concurrent-safe would. Scale out with more processes behind any HTTP balancer.

The thing to size for is memory, not CPU: each process holds its own copy of the reader, so four workers with the 4B reader is roughly 40 GB. Four workers with the 0.8B is about 16 GB. That ratio is usually what decides which reader a service ships with.

Note

A guard in front of your model wants this surface.

A gateway that checks requests before forwarding them is the most common reason to run this server: the guard sheet is one POST /v1/decide away from any language, and nothing leaves your infrastructure. The prompt-injection guard recipe has the sheet and the measured numbers.

A client with no torch in it#

If you are using the LangChain integration, pass url= instead of model= and every call goes over HTTP to this server. The client process never imports torch, which matters for a web worker or a lambda.

python

from vegaml.integrations.langchain import VegaDecision

decide = VegaDecision(url="http://decisions.internal:8000", questions=questions)