Build the evaluation harness before you choose the model

The standard sequence for an enterprise AI project is: pick a model, build a prototype, demo it to stakeholders, then attempt to establish whether it is actually good enough to deploy. By the time that question gets asked seriously, the prototype has a sponsor, a name and a slide in a board pack.

We run it the other way round, and it is the single highest-leverage change we have made to how we deliver AI work.

What the harness has to contain

Before any model is chosen, we build a dataset of real cases with expert-adjudicated correct answers. Not synthetic examples — actual historical cases from the client's own records, labelled by the people who currently do the task. For the prior-authorisation work we describe in our Cascade case study, that was 1,400 adjudicated packets. It took six weeks and three clinicians.

Then we define what "correct" means in a way that can be computed, which is usually harder than gathering the data. For extraction tasks it is field-level precision and recall. For generation it is a rubric applied by a second model and spot-audited by humans. For decisions it is agreement with the expert panel, weighted by the cost of each error type — because a false negative and a false positive rarely cost the same, and averaging them hides the thing you care about.

If you cannot compute the quality of an output, you do not have a project. You have a demo.

What it changes

Model selection stops being a debate. You run the candidates through the harness and read the numbers. On our last three engagements the model the client expected to win did not, twice.

Prompt and retrieval changes stop being guesswork. Every change is measured. We routinely find that retrieval quality dominates model quality by a wide margin, which redirects effort towards the data work where it belongs.

The governance conversation gets much shorter. Compliance functions are not obstructive; they are being asked to approve something with no evidence. Hand them per-error-type performance on their own historical cases, with a documented adjudication process, and the conversation changes character entirely.

You find out early when the answer is no. Roughly a third of the use cases we harness turn out not to clear the bar at acceptable cost. Discovering that in week six for the price of an evaluation dataset is a good outcome, not a failed project.

The objection

The objection is always that building the harness delays the demo by six weeks. It does. It also means the demo is a deployment candidate rather than a prototype, and it removes the six months that usually follow a successful demo while everyone works out whether the thing is safe.

Six weeks up front. We have not once regretted it.