Docker / docs / embedding

The HTTP server as a container, with the checkpoint cached in a volume so a restart does not re-download it.

The image wraps vegaml-serve. The checkpoint downloads on first request and persists in a volume, so restarts do not re-download it.

bash

docker compose up                      # http://localhost:8000

# or, by hand
docker build -t vegaml .
docker run -p 8000:8000 -v vegaml-hub:/root/.cache/huggingface vegaml

VEGAML_MODEL picks the checkpoint: empty for the default 0.8B, 4b for the 4B reader, or any Hub id.

The volume is not optional#

Without -v on the HuggingFace cache, every container start downloads the reader again: 1.77 GB for the default and 9.34 GB for the 4B. In a restart loop that is a bandwidth bill and a very confusing latency graph.

Watch out

Set a health check that waits for the model.

GET /health returns {"status": "ok", "model_loaded": false} before the first request has loaded anything. An orchestrator that treats a 200 as ready will send real traffic into the cold load. Either check model_loaded, or start the server with --preload so it does not bind until the model is resident.

The image is CPU-only on purpose#

It exists so the HTTP API runs anywhere a container does. A GPU host is better served by pip install "vegaml[serve]" against its own torch build, because the right torch wheel depends on the host's CUDA, and baking one into an image makes the image wrong somewhere else.

On CPU the default reader decides in tens of milliseconds, which is fast enough for a gateway. If you are considering a GPU for a decision workload, measure first: the read is small and the engine is 14.3M parameters.