Long context and images / docs / embedding

73,728 tokens, prefix caching that makes twelve questions about one contract cost roughly one read, and decide_image for pixels.

max_len is 73,728 tokens, and the architecture is built for that rather than merely permitting it. Three things make a long state behave.

Read-once prefix caching#

A state of at least 4,096 tokens is encoded once; every question about that state continues from a copy of that prefix. Twelve questions about one 73k-token contract cost roughly one read, not twelve.

python

contract = open("msa.txt").read()        # 70,000 tokens

out = v.decide(contract, {
    "governing_law":  {"type": "choice",  "instructions": "Which law governs?",
                       "criteria": {"england": "...", "delaware": "...",
                                    "india": "...", "other": "..."}},
    "auto_renew":     {"type": "boolean", "instructions": "Does it renew automatically?"},
    "liability_risk": {"type": "score",   "instructions": "How exposed are we?",
                       "criteria": ["capped at fees", "capped at a multiple",
                                    "uncapped for some heads", "uncapped"]},
    # ... nine more
})

out["n_tokens"]      # one read's worth, not twelve
out["latency_ms"]

This is the single biggest reason to put several questions on one sheet rather than calling decide once per question. The sheet is free; the reading is not.

A memory reader for what sits late in a document#

Long states are cut into 16 pieces, mean-pooled at the deepest feature layer, and fed to the engine alongside the pooled spans. Evidence in the last paragraph of a long contract still reaches the decision rather than being crowded out by the first page.

No silent truncation#

An input that will not fit is refused, never quietly shortened. This matters more than it sounds: a model that trims your document and answers anyway gives you a confident answer about a document you did not send. If you want truncation, do it yourself, on purpose, and then you know it happened.

Note

Truncate deliberately when you are comparing models.

If you are scoring several guards or readers against each other, cut every input to the same length before any of them see it. Our own guard benchmark truncates every case to 1,200 characters for exactly this reason: a hosted model that silently handles more text than a local one with a 1,024-token cap is not being compared, it is being flattered.

Images#

python

pip install "vegaml[vision]"

python

from PIL import Image

v.decide_image(Image.open("invoice.png"),
               {"kind": {"type": "choice",
                         "instructions": "What kind of document is this?",
                         "criteria": {"invoice": "a bill",
                                      "receipt": "proof of payment",
                                      "form": "a document to be filled in",
                                      "other": "none of these"}}})

The backbone is multimodal and its vision encoder is frozen with the rest, so a picture takes the place of the state text in the same prompt and the same spans are pooled. image may be a PIL image, a path, or an HxWx3 numpy array.

Answers come back in the same shape as decide, calibration and all: prediction_set, abstain, unbound, settle_steps and energy are present exactly as on the text path.

Two honest limits. decide_image always runs the engine and marks every answer readout: "engine", because a fitted head reads text features and has nothing to say about pixels. And the published benchmark figures are text; the image path is functional but not covered by them.

To measure what the picture actually contributed, decide_image_text answers the same questions through the same path with the image withheld. If the two agree, the decision was not coming from the pixels.