Evals

Automated tests for AI — a fixed set of tasks with expected results, run every time something changes.

Builder path · 4 of 6

What it is

Evals measure how well an AI system does its job — accuracy, safety, cost, speed — on a fixed set of examples. Because models and prompts change often, evals work like regression tests in software.

Example

Before switching to a new model, a set of typical customer questions is run through both versions and the answers are compared.

What can go wrong

Without evals, a model update or a small prompt change can quietly make results worse.

How to secure it

Keep a test set that reflects real use — including attack attempts — run it on every model or prompt change, and track the results over time.

Related service: Applied AI