AI and agents
Evals
Evals are repeatable tests that measure how well an AI system does its job, by running it on a fixed set of real examples and scoring the outputs against what a good answer looks like.
How it works in practice
Ordinary software tests check that the same input gives the same output. AI output varies, so evals score it instead: was the right field extracted, was the email routed to the right team, did the answer cite the right policy. Some checks are exact, some use a rubric, and some use a second model as a grader with people checking a sample.
Evals are how you know whether a prompt change, a new model or a cheaper model made things better or worse. We build an eval set from the client’s own past cases during the build, run it before every change and keep it running after go-live on a sample of live traffic.
A worked example
Before switching an invoice extraction agent to a cheaper model, a finance team runs both models on 200 past invoices. The cheaper model gets supplier and total right as often, but misreads VAT on 9 invoices, so VAT extraction stays on the larger model.
Examples in this glossary describe typical builds for companies of 1 to 1,000 people. They are not named client results.















