Questions and answers

A request is a state plus a mapping of question ids to questions. The question sheet is the part you write, and it is the part that decides how good the answers are. There is no prompt engineering here in the usual sense: there is a short instruction and, for most types, a definition per option. The definitions are what the model reads.

questions = {
    "team":     {"type": "choice",  "instructions": ..., "criteria": {...}},
    "churn":    {"type": "boolean", "instructions": ...},
    "severity": {"type": "score",   "instructions": ..., "criteria": [...]},
}

Every id in that mapping comes back as a key under answers. Ids are yours; keep them stable, because they are what your code reads and what calibration and fitted heads are keyed by.

choice#

Pick one option from a set you define. criteria maps each option to the words that define it.

{"team": {"type": "choice",
          "instructions": "Which team handles this?",
          "criteria": {"billing": "payments, invoices and refunds",
                       "technical": "bugs and outages",
                       "sales": "pricing and plans"}}}

A bare list is accepted ("criteria": ["billing", "technical"]) and means options with no definitions. It is almost always worse. The option text is read by the model, so one line per option is the cheapest accuracy you will ever buy.

The answer carries choice, the winning option name, and probs keyed by the same names.

boolean#

Does the statement hold? The answer's boolean field is the probability of yes, which is more useful than a bare True because you can threshold it where your costs say to.

{"churn": {"type": "boolean",
           "instructions": "Does the customer threaten to leave?",
           "criteria": {"true": "shows intent to cancel",
                        "false": "no such signal"}}}

Criteria are optional on a boolean and sharpen both sides when the statement is ambiguous. Give the false side real words too: “no such signal” is doing work, because it tells the model what the absence looks like.

score#

An integer level against ordered criteria. criteria is a list, one description per level, lowest first.

{"severity": {"type": "score",
              "instructions": "How severe is the problem?",
              "criteria": ["cosmetic",
                           "degraded for some users",
                           "an outage"]}}

The answer carries score, the winning level as an index from 0, and expected_score, the probability-weighted level. For sorting a queue, use expected_score: it separates a confident 2 from a coin flip between 1 and 3, which both report score == 2.

Writing criteria that work#

Four rules, learned the expensive way.

Describe the property, never the string

A criterion that quotes the text it expects to see is beaten by the first input that rephrases. Say what makes an option true, not what it looks like. The prompt-injection guard is the sharpest example: its block criterion describes an attempt to subvert the assistant, and deliberately contains no attack names and no quoted phrases.

Make the options mutually exclusive and jointly exhaustive

If two options can both be true, the probabilities split between them and the confidence drops for a reason that has nothing to do with the state being unclear. If no option fits, something still wins. Add "other" with a real definition, and treat traffic landing there as the signal it is.

Write the instruction as a brief, not a prompt

One sentence of what decision is being made and on whose behalf. “Which team handles this?” is enough. Role-play, politeness and threats do nothing, because nothing is being generated: the instruction is read, pooled, and turned into coordinates.

Put per-row variation in the question, not only in the state

If what distinguishes your examples sits in the instruction rather than in the state, as it does in most intent classification, pass the question per example when you fit, or every example gets an identical prompt and the head is fitted on noise:

examples = [(state, labels, {"intent": dict(question,
                                            instructions="...this row's text...")}),
            ...]

Every field on an answer#

a = out["answers"]["team"]

a["choice"]            # the picked option
a["probs"]             # {option: probability}, calibrated, sums to 1
a["confidence"]        # the top probability
a["prediction_set"]    # options the conformal guarantee keeps at your alpha
a["abstain"]           # True: confidence is under the risk threshold
a["unbound"]           # True: the state sat off the calibrated map
a["settle_steps"]      # integrator steps taken
a["energy"]            # how deep the chosen well was
a["readout"]           # 'engine' or 'ttt': which path answered

prediction_set is None rather than a list when the checkpoint has no conformal threshold for that type and alpha, so test it for truthiness rather than indexing into it blind.

Two readouts, one set of features#

mode picks how the pooled features become an answer. The default is "auto", and with nothing fitted "auto" is the engine, flags and all.

The detail that bites: a head-served answer has no abstain bit. See Fitting a head for what to do about that.