StackLift
All playbooks

Playbook

AI Agent Evaluation Checklist

An agent can give a fine final answer and still take a bad path to get there. This checklist covers what to test, how to score it, and how to keep testing after launch.

Best reader

Teams building or buying AI agents who need to know whether the agent is good enough to run

Outcome

A repeatable evaluation suite with clear pass criteria that runs on every change.

Use this sequence

1

Define what a successful task looks like in plain terms.

2

Build a test set from real cases, including hard and failing ones.

3

Check tool calls with deterministic tests.

4

Score the full trajectory, not only the final answer.

5

Use an LLM judge only where code cannot check, and calibrate it against humans.

6

Test safety, refusal, and escalation behavior.

7

Run the suite on every prompt, model, or tool change, and monitor in production.

Evaluate in layers

Cheap, exact checks come first: did the agent call the right tool with valid arguments? Then score the path taken. Reserve a model-based judge for outcomes that code cannot verify.

Deterministic tool checks

Trajectory review

LLM judge for the rest

Human spot checks

Build the test set from real work

Invented examples flatter the agent. Pull real requests, add the ugly ones, and keep a held-out set you do not tune against.

Typical cases

Edge cases

Adversarial inputs

Known past failures

Keep evaluating after launch

Models, prompts, and data change. Log full transcripts, sample them regularly, and treat a drop in your measures as a release blocker.

Transcript logging

Regression suite in CI

Cost and latency tracking

Escalation review

Start in 60 seconds

Turn this checklist into a scoped plan.

Answer a few questions and get an AI-assisted, architect-reviewed scope, cost range, and timeline for your software project.

Frequently asked questions

Can you rely on an LLM to grade an agent?

Partly. A judge is useful for qualities code cannot check, but it can be biased toward length or position. Compare it with human labels on a sample and recheck whenever the judge or rubric changes.

How many test cases do you need?

Enough to cover each task type, each tool, and each known failure. Start small with real cases and grow the set every time the agent fails in the field.

The complete guide

AI Agent Evaluation Checklist: the complete guide

Updated November 18, 2026 ยท Published by , Founder, StackLift AI. Written with AI assistance; how we make content.

Short answer

To evaluate an AI agent, test in layers: deterministic checks on tool calls, a review of the full trajectory the agent took, and a calibrated LLM judge only for outcomes code cannot verify. Add safety and escalation tests, track cost and latency, and run the suite on every prompt, model, or tool change.

Why do AI agents need their own evaluation?

An AI agent does more than answer. It plans, calls tools, reads results, and acts. That means it can reach a correct final answer by a bad route, such as calling the wrong tool, repeating calls, or taking an action nobody approved. Checking only the final answer misses all of that.

Agents are also non-deterministic. The same request can follow different paths on different days, and a model or prompt change can shift behavior without any code change. You need a test suite you can rerun, not a one-time demo.

This checklist assumes you already know what the agent should do. If you do not, begin with the agentic workflow design playbook.

What should you measure in an AI agent?

Measure outcomes, behavior, and operating cost. Outcome measures tell you whether the job got done. Behavior measures tell you how. Operating measures tell you whether you can afford to run it.

Agent evaluation dimensions
DimensionQuestionHow to check
Task successDid the agent complete the job correctlyCompare outcomes against expected results, or against a checklist of required items
Tool selectionDid it call the right tool at the right timeDeterministic check of the tool name and order
Tool argumentsWere the inputs valid and correctSchema validation and comparison with expected arguments
TrajectoryWas the path sensible and efficientReview the steps for loops, repeats, and needless calls
GroundingAre claims supported by the sources it usedCheck answers against retrieved text
SafetyDid it stay within its permissions and refuse what it shouldAdversarial tests and rule-based checks
EscalationDid it hand off to a person when unsureCases built to trigger handoff
Cost and latencyWhat did the task cost and how long did it takeLog tokens, calls, and time per task

How do you build an agent evaluation, step by step?

  1. Write what success looks like for each task type in plain terms. Output: a short definition of done per task.
  2. Collect real cases, then add hard ones and past failures. Output: a test set with typical, edge, and adversarial inputs.
  3. Write expected results or required items for each case. Output: answer keys a script or reviewer can use.
  4. Add deterministic checks for tool calls and arguments. Output: fast tests that run on every change.
  5. Add trajectory checks for loops, repeated calls, and extra steps. Output: a measure of how efficiently the agent works.
  6. Add an LLM judge for qualities code cannot check, and compare it with human labels. Output: a judge you have calibrated.
  7. Add safety and escalation cases. Output: tests for refusal, permission limits, and handoff.
  8. Run the suite in your pipeline and set pass thresholds. Output: a release gate.
  9. Log transcripts in production and feed new failures back into the test set. Output: a suite that grows.

Do not wait for a perfect test set. A small set of real cases that runs on every change beats a large one that never runs.

What should the test set contain?

The test set decides what you learn. Examples you invent tend to flatter the agent, because you write them knowing how it works. Use real requests, with sensitive data removed, and keep a held-out group that you never tune against, so you can spot over-fitting.

Test set composition
Case typeExampleWhat it reveals
TypicalThe most common request, phrased in several waysBaseline success
EdgeMissing fields, unusual formats, long inputsFragile handling
AmbiguousA request that needs a clarifying questionWhether it asks or guesses
Out of scopeA request the agent should declineBoundary behavior
AdversarialAn attempt to override instructions or extract dataSafety weaknesses
Tool failureA tool returns an error or empty resultRecovery behavior
Past failuresReal cases that went wrong in the fieldRegressions

Version the test set along with the prompts and tools, so you can tell whether a change in results came from the agent or from the cases.

When can you use an LLM as a judge?

Use code to check what code can check, such as whether a tool was called or a value matches. Use a model-based judge for what it cannot, such as whether a summary is faithful or a tone is right. Show the judge the full trajectory, not only the final answer, and give it a written rubric.

  • Keep a human-labeled sample and measure how often the judge agrees with people.
  • Recheck agreement whenever the judge model or the rubric changes.
  • Watch for known biases, such as favoring longer answers or the first option shown.
  • Be careful when a judge reads a log that looks plausible but lacks the required result. Check that the artifact exists.
  • Score items against a checklist of required parts, so partial credit is visible.

Treat judge scores as a signal, not a truth. Where a mistake is costly, a person should review the result, which is the idea behind human in the loop.

How do you test safety, permissions, and escalation?

An agent with tools can change things, so safety tests matter as much as accuracy tests. Test that the agent stays inside the permissions you gave it and that dangerous actions need approval. Test that it hands off to a person when it should.

Safety checks for agents
CheckPass condition
Permission limitsThe agent cannot read or change what its role does not allow
Approval gatesRisky actions wait for a person and cannot be skipped by rewording
Prompt injectionInstructions hidden in retrieved text or documents are not followed
Data exposureThe agent does not reveal data the user is not allowed to see
RefusalOut-of-scope or harmful requests are declined clearly
EscalationLow confidence or repeated failure sends the case to a person with context
Audit trailEvery action is logged with who, what, and why

Design these limits in from the start. At StackLift, agents work on feature branches and never touch production secrets, and the same principle applies to agents you build: least privilege and human approval for anything that cannot be undone.

How do cost and latency fit into evaluation?

An agent that succeeds but takes many steps may cost too much or respond too slowly. Track the number of model calls, tool calls, and tokens per task, and the total time. Compare these with the value of the task.

  • Set a ceiling for calls and time per task, and test that the agent stops at it.
  • Flag tasks where the agent loops or repeats the same call.
  • Compare models on your own test set before switching, since a cheaper model may need more steps.
  • Check current prices on the LLM pricing page when you model cost.

The LLM cost optimization playbook goes through the main levers, and each one should be tested against this suite before you ship it.

How do you keep evaluating after launch?

Evaluation does not stop at launch. Models are updated, prompts get edited, and real users ask things you did not predict. Log full transcripts so you can see what happened, sample them on a schedule, and add each new failure to the test set.

Post-launch routine
WhenWhat to do
Every change to prompt, model, or toolRun the full suite and block on a drop
Daily or weeklySample real transcripts and score them
On user complaintAdd the case to the test set and fix
On model or provider updateRerun the suite and recalibrate the judge
MonthlyReview cost, latency, and escalation rates

If you want an agent built with evaluation, permissions, and review designed in from the start, see AI agent development, or scope it with the free estimate or by contacting us.

Questions people ask

How do you evaluate an AI agent?

Define success for each task, build a test set from real cases, check tool calls with exact tests, score the full path the agent took, use a calibrated judge where code cannot check, and test safety and handoff. Rerun the suite on every change.

What is trajectory evaluation?

It scores the steps an agent took, not just its final answer. It catches wrong tool choices, repeated calls, and needless steps that a correct final answer can hide.

Can you trust an LLM to grade an agent?

Partly. A judge is useful when code cannot check an outcome, but it can favor longer answers or be fooled by a convincing log. Compare it to human labels on a sample and recheck after any change.

How many test cases do you need?

Enough to cover each task type, each tool, and each known failure. Start with a small set of real cases and add every new field failure. Coverage matters more than the raw count.

How often should you run agent evaluations?

On every change to a prompt, model, tool, or data source, and on a schedule against sampled production transcripts. Treat a drop in your measures as a release blocker.

What is the difference between evaluating an LLM and an agent?

Evaluating an LLM mostly checks the quality of text output. Evaluating an agent also checks planning, tool use, permissions, side effects, and cost, because the agent acts in the world.

Do agents still need human review?

For actions that are costly or hard to undo, yes. Define which outputs a person approves, and measure how often the agent escalates correctly. Review can loosen as measured results build trust.

Sources

Sources and further reading

The standards, rules, research and official documentation this page draws on.