ContactBook a call

AI and agents

Evals

Evals are repeatable tests that measure how well an AI system does its job, by running it on a fixed set of real examples and scoring the outputs against what a good answer looks like.

How it works in practice

Ordinary software tests check that the same input gives the same output. AI output varies, so evals score it instead: was the right field extracted, was the email routed to the right team, did the answer cite the right policy. Some checks are exact, some use a rubric, and some use a second model as a grader with people checking a sample.

Evals are how you know whether a prompt change, a new model or a cheaper model made things better or worse. We build an eval set from the client’s own past cases during the build, run it before every change and keep it running after go-live on a sample of live traffic.

A worked example

Before switching an invoice extraction agent to a cheaper model, a finance team runs both models on 200 past invoices. The cheaper model gets supplier and total right as often, but misreads VAT on 9 invoices, so VAT extraction stays on the larger model.

Examples in this glossary describe typical builds for companies of 1 to 1,000 people. They are not named client results.

See which of these apply to your team

On a discovery call we map the manual work in your team and show you which of these would change it first.

Book a discovery call

Cookie preferences

Choose what this site may use on your device. You can change this at any time from “Cookie settings” in the footer.

Strictly necessary

Keep the site working and remember the choices you make here. These cannot be switched off, and none of them track you.

Always on
What this includes (4)
NameProvider PurposeDuration
sbd-consentThis site Remembers your cookie choices12 months
Vercel Web AnalyticsVercel Inc. Counts page views in aggregate. Sets no cookie and stores nothing on your deviceNone
ticker-dismissed, sbd-note-offThis site Remembers that you closed a notice or language tipUntil the tab closes
sbd-voice-resume, sbd-voice-prefillThis site Carries a voice conversation across pages, only if you start oneUntil the tab closes

Analytics

Tell us which pages are read and where visitors leave, so we can improve them. IP addresses are anonymised and nothing is used for advertising.

What this includes (2)
NameProvider PurposeDuration
_gaGoogle Analytics (Google LLC) Distinguishes visitors with a random ID13 months
_ga_FPXDMQPRV1Google Analytics (Google LLC) Keeps the session state for this site13 months

Functional

Embedded services that set their own cookies. Without them the site still works; you get a link to open the service instead.

What this includes (1)
NameProvider PurposeDuration
Booking calendarCal.com Inc. Shows live availability and books the call on the booking pageSet by Cal.com

Advertising

Not used. This site has no advertising, retargeting or social media tracking, and we never sell data.

Not used

Questions about your data: hello@smallbaydigital.com