Evals are how you know the machine is allowed to decide.
PRAXIS says what to delegate. Evals are the evidence that the delegation holds: before launch, after launch, and every time the model, the prompt or the data changes.
This is a method page: what PRAXIS implies once it meets a real model. It makes no claim about any specific engagement.
Why evals belong to the product leader
Most teams treat evaluation as a technical chore: a benchmark score, a leaderboard position, a test suite engineering will get to. That is how products end up shipping on a good demo and a feeling.
An eval is a product decision written down in a form a machine can be tested against. What counts as a good answer? Which mistakes are tolerable? Which must be impossible? They are product questions, the same ones PRAXIS asks about the delegation threshold, except that now they must survive contact with a system that is probabilistic, changes without warning and fails in ways nobody listed.
So the owner of the eval is the owner of the product. Engineers build the harness. The product leader decides what it measures and what happens when the number is bad.
From judgement to evidence
Every PRAXIS concept has an operational twin. If it cannot be tested, it stays a principle on a slide.
An autonomy map per decision: what the system may do alone, what needs a person, what is out of bounds. Each level gets its own pass criteria.
Autonomy map + a bar per decision
An error taxonomy weighted by severity. Visible, understandable, reversible failures are budgeted. Opaque, irreversible, silent ones are not.
Severity-weighted scorecard
Red lines become hard gates. One violation blocks the release, whatever the average score says.
Zero-tolerance test set
Two questions: "was the model right?" and "could the person notice, understand and correct it?". Escalation and override quality are measured.
Hand-off and override metrics
Behaviour under ambiguity: does the system abstain, ask or hedge when it does not know, or does it improvise with confidence?
Calibration and abstention cases
The behaviour spec, written before the dataset: what the system must do, must not do, and does when it is unsure.
Behaviour spec (the PRD chapter that is usually missing)
The eval loop
Six moves, in this order. The order matters: most failed eval efforts start at step two and never write step one.
- 01
Specify the behaviour
Before any dataset, write in plain language what the system must do, must not do, and does when it is unsure. One spec per decision it takes. This is where the delegation threshold is actually set.
- 02
Build the dataset from reality
Start from real cases and real failures: support tickets, edge cases, places where experts disagree. Synthetic optimism is a poor starting point. A small, curated set beats a large, generic one. Include the hard cases on purpose, and version the set like code.
- 03
Choose graders by what they must catch
Code-based checks where the answer is checkable. Human experts where judgement is the point. An LLM as judge for scale, but only after it has been calibrated against human labels and its own errors are known. An unchecked judge is just another model you are trusting.
- 04
Set gates on top of scores
Averages hide the failures that matter. Define thresholds per severity class and hard fails for red lines. Decide in advance what blocks a release, what triggers a rollback, and who can overrule it.
- 05
Run it on every change
A model swap, a prompt edit, a retrieval tweak, new data: each one is a release. Run the regression suite on all of them, compare versions, and track cost and latency next to quality. Quality bought at ten times the cost is a trade-off someone has to decide on.
- 06
Keep evaluating in production
Offline evals tell you the system is ready. Production tells you it is still right. Sample real traffic, read it, and turn every new failure into a test case. Treat user corrections, escalations and drift as signals.
The scorecard I would want on my desk
A short set of measures, each tied to a decision someone has the authority to make.
- Task success, by decision type
- Split by decision type. A system can be excellent at the easy 90% and dangerous at the rest.
- Severity-weighted error rate
- Errors counted by what they cost the person, plus a separate count of red-line violations. The target for that count is zero.
- Groundedness
- For systems that answer from sources: does every claim trace back to something the system was actually given?
- Calibration and abstention
- Does confidence track correctness? Does the system say "I do not know" when it should, and only then?
- Escalation quality
- Precision and recall of hand-offs to a person, and how long a person takes to detect and fix a wrong output.
- Grader agreement
- Agreement between human experts, and between the judge model and the humans. If experts disagree, the spec is not finished.
- Cost and latency per resolved task
- Quality without its price is half a metric. Track both, per version.
- Drift
- How all of the above move over time and across model, prompt and data changes.
Where evals go wrong
The recurring ways an evaluation effort produces confidence without evidence.
- Optimising for the benchmark
- Public benchmarks measure someone else's task. Your users are running yours.
- One blended accuracy number
- It hides the rare, severe failures, which are exactly the ones that end up in an incident report.
- A judge model nobody checked
- Scaling grading with an LLM is fine, as long as you have measured its errors first.
- A static test set
- It stops resembling production the day you ship. If failures do not flow back into the set, the eval decays.
- Evaluating the model when the failure lives in the product
- Retrieval, permissions, the interface and the hand-off fail too. The unit under test is the whole system the person meets.
- A dashboard with no decision attached
- If nobody has the authority to block a release or roll one back, the eval is decoration.
- Human review as the fallback
- When the person is asked to catch what the system does not flag, the product has moved its responsibility onto the user. Review has to be designed.
Questions
- What is an AI eval?
- A repeatable test of whether an AI system behaves as specified, run on cases that represent real use and scored against explicit criteria by graders: code, human experts or calibrated models.
- Who should own AI evals: product or engineering?
- Product owns what is measured and what happens when it fails. Engineering owns the harness. If nobody with product authority owns the spec, the eval ends up measuring whatever was easy to measure.
- Can an LLM judge replace human evaluators?
- It can scale grading in narrow, well-specified cases, once it has been calibrated against expert labels. It cannot decide what good means. That stays human: the eval spec is where the delegation threshold is set.
- How big does an eval set need to be to start?
- Smaller than most teams expect. A few dozen carefully chosen real cases, including the hard ones, are more useful than thousands of generic ones. The set should grow every time production surprises you.
- When are evals done?
- Never. Models, prompts, data and users all change. Evals are a permanent part of the product, in the same way monitoring is a permanent part of operations.
Shipping an AI feature and unsure how you will know it works?
Let us define what good means, what must never happen, and what blocks a release, before the model decides anything for your users.