Playbook
AI Agent Evaluation Checklist
An agent can give a fine final answer and still take a bad path to get there. This checklist covers what to test, how to score it, and how to keep testing after launch.
Best reader
Teams building or buying AI agents who need to know whether the agent is good enough to run
Outcome
A repeatable evaluation suite with clear pass criteria that runs on every change.
Use this sequence
Define what a successful task looks like in plain terms.
Build a test set from real cases, including hard and failing ones.
Check tool calls with deterministic tests.
Score the full trajectory, not only the final answer.
Use an LLM judge only where code cannot check, and calibrate it against humans.
Test safety, refusal, and escalation behavior.
Run the suite on every prompt, model, or tool change, and monitor in production.
Evaluate in layers
Cheap, exact checks come first: did the agent call the right tool with valid arguments? Then score the path taken. Reserve a model-based judge for outcomes that code cannot verify.
Deterministic tool checks
Trajectory review
LLM judge for the rest
Human spot checks
Build the test set from real work
Invented examples flatter the agent. Pull real requests, add the ugly ones, and keep a held-out set you do not tune against.
Typical cases
Edge cases
Adversarial inputs
Known past failures
Keep evaluating after launch
Models, prompts, and data change. Log full transcripts, sample them regularly, and treat a drop in your measures as a release blocker.
Transcript logging
Regression suite in CI
Cost and latency tracking
Escalation review
Start in 60 seconds
Turn this checklist into a scoped plan.
Answer a few questions and get an AI-assisted, architect-reviewed scope, cost range, and timeline for your software project.
Frequently asked questions
Can you rely on an LLM to grade an agent?
Partly. A judge is useful for qualities code cannot check, but it can be biased toward length or position. Compare it with human labels on a sample and recheck whenever the judge or rubric changes.
How many test cases do you need?
Enough to cover each task type, each tool, and each known failure. Start small with real cases and grow the set every time the agent fails in the field.