decision

Measure the host before building the model

2026-10-14 · UX-192

The question

A small model trained to say what a healthcare payload is — which standard and version, which workflow, which direction, which actors, where it is going — failed its own holdout. What should the next build be?

Constraints

  • The frozen model failed a small holdout. There was no body of expert-corrected outcomes to train against or measure with, so more tuning would have been tuning against noise.
  • The first attempt spent its effort on study consent, signed publication artifacts and release controls before anyone but the author could run it.
  • People already ask general-purpose AI hosts this question, usually by pasting the payload in. A real payload carries identifiers, clinical values and free text, and once pasted it is subject to the host provider's policy, not yours.
  • Synthetic data only. Nothing in the build may need a real payload to work.
  • One maintainer. Whatever ships has to be runnable by someone else from a clean clone.

Alternatives considered

  • Keep refining the model.

    Rejected — It failed its holdout, and the missing ingredient was not the model. It was labelled outcomes from someone who knows the answer. A better model trained on the same absence would have failed the same way, later.

  • Build a validator.

    Rejected — A validator answers "is this conformant?", which is a different question from "what is this, and where is it going?". Good validators already exist, and this site already runs one in the Learning Lab.

  • Let the AI host read the raw payload.

    Rejected — It is the easiest path and has the most exposure. It also makes the result unrepeatable: the host's answer depends on values the evaluation record is not allowed to keep.

  • Accept free-text answers with confidence scores.

    Rejected — Prose can carry values back out, and a fluent paragraph that is wrong in one detail reads the same as one that is right. Uncalibrated confidence numbers look like evidence and are not. Neither can be compared across hosts or runs.

  • Measure the host first: a model-neutral loop that shows it only an approved structural summary, takes closed-code answers, and has an expert correct each one.

    Chosen.

The decision

Build the instrument before the model. Payload Intelligence is a local workbench. Deterministic code reads a synthetic payload inside a one-use browser Worker. A person approves a value-minimized structural summary — every field the host will see, shown in full. An MCP-capable AI host (Codex or Claude Desktop) answers five axes from a closed, versioned taxonomy. An interoperability SME then accepts, corrects or marks each answer not scorable. The output is a sanitized record of where the host was right, wrong or unable to tell — the labelled outcomes the first attempt never had.

What the first attempt got wrong

The order of work, more than any single choice. The first attempt built study consent, signed publication artifacts and release lifecycle controls around a model before the experience could be handed to anyone. When the model failed its holdout, all of that machinery was protecting a result that did not exist.

Neither the model nor the study machinery carried forward. The new repository's inventory check fails the build if study or model-pack artifacts reappear, so the same mistake cannot return by accident.

Why closed codes are the decision that matters

The host never writes prose. It picks from fixed lists — FHIR_R4, HL7_V2_5_1, PRIOR_AUTHORIZATION, REQUEST, PAYER_SYSTEM and so on. Each answer must cite evidence references the case actually offered. "I can't tell" is an explicit code, and it must carry the conflict that made it unknowable. Probabilities, rationale, URLs and any field the schema does not name are rejected outright.

That gives up a lot of what a model can say. It is also the only way the output becomes countable: every correction can be compared with every other, across hosts and across runs. Without that, an expert's corrections are opinions, not data.

Where it stands, 2026-10-14

What is proven. Executable tests cover the boundary and the loop:

  • raw input never reaches the coordinator or the host;
  • the host gets exactly three tools and can only read an approved case and submit an answer;
  • nothing is accepted by default;
  • a correction must actually differ from the host's answer.

A fresh-clone reproduction outside the author's machine passed the unit, browser and policy checks.

An outside review also found a defect the suite had missed. Launching the lab a second time quietly cut the running lab off from its AI host, while its screen still said it was ready. It was fixed with a regression test before this record was published.

What is not proven.

  • No real host has completed the loop.
  • No one other than the author has run it.
  • There is no model-quality result yet. Producing that evidence is the point of the workbench.

The preview makes no conformance, compliance or production claim, and uses synthetic data only.

The part worth keeping

  • When a model fails, ask what it was missing before building a better one. Here it was expert-corrected outcomes, and those need an instrument, not a model.
  • Build the thing someone else can run before building the controls around it.
  • Make the answer format closed before collecting answers. Free text feels richer and cannot be counted.
  • Show the person the full summary they are approving, not a description of it.

Where this is canonical

docs/backlog/cards/UX-192.md — The engineering record is canonical in the public Payload Intelligence repository: ADR 0001 and docs/WRITEUP.md in github.com/ImOnFHIR/payload-intelligence. This record is a dated snapshot.

Related

Back to Decisions