StackLift
All terms

Glossary

Agent evaluation

Testing whether an AI agent completes tasks correctly, safely and efficiently, using repeatable scenarios.

Agent evaluation is the practice of testing an AI agent on a fixed set of realistic tasks to measure whether it reaches the right outcome, uses tools correctly, stays within its limits and does so at acceptable cost and speed. It is repeated after every change to the prompt, tools, data or model.

Agents are harder to test than ordinary software because the same input can lead to different paths. Good evaluation therefore checks outcomes and behavior across many cases, not a single run.

A test set (often called an eval set) is built from real cases, including awkward ones and attacks. Results are scored by exact checks where possible, by human review, and sometimes by another model acting as a judge.

Evaluation is how you learn that a model upgrade or prompt tweak made things worse. Without it, quality drifts unseen.

Start in 60 seconds

Put this to work on your project.

Get an AI-assisted, architect-reviewed scope, cost range, and timeline. Free, no login.

The complete guide

Agent evaluation: the complete guide

Updated November 18, 2026 · Published by , Founder, StackLift AI. Written with AI assistance; how we make content.

Short answer

Agent evaluation is testing an AI agent on a repeatable set of realistic tasks to see whether it reaches the right outcome, uses its tools correctly, stays inside its limits and does so at acceptable cost and speed. You repeat the tests after every change to the prompt, tools, data or model.

What is agent evaluation?

Agent evaluation is how you find out whether an AI agent actually does its job. You build a set of realistic tasks, run the agent on them, and score what happened: the final answer, the steps it took, the tools it called and what it touched along the way.

It is the agent version of a test suite, with one big difference. Ordinary software gives the same output for the same input. An agent built on a large language model can take a different path each time, so you judge behavior across many cases and many runs, not a single pass.

People often call the task set an eval set, and the scored run an eval. You will hear both words used for the whole practice.

Why does agent evaluation matter?

An agent that looks good in a demo can fail in ways nobody saw: it picks the wrong tool, loops, invents a fact, or follows an instruction hidden in a document. A demo shows the best case. An evaluation shows the spread.

  • It tells you whether the agent is ready for real users, using evidence instead of opinion.
  • It catches regressions when you change a prompt, swap a model, edit a tool or update the data the agent reads.
  • It lets you compare options fairly, for example a cheaper model against the current one on the same cases.
  • It gives risk, legal and security reviewers something concrete to look at.
  • It sets the quality bar in writing, so a launch decision is not an argument.

Without evaluation, quality drifts unseen. You learn about problems from customer complaints, which is the most expensive way to learn.

How does agent evaluation work, step by step?

  1. Define what good looks like for this agent in plain terms: the outcome it must reach, the actions it may take and the things it must never do.
  2. Collect real cases from tickets, emails, documents or logs, and remove personal data where it is not needed.
  3. Add awkward cases: missing information, conflicting sources, unusual phrasing, requests outside its scope and attempts to push it off its rules.
  4. Write the expected result or the scoring rule for each case, such as the correct field value, the right tool call or a refusal.
  5. Run the agent on every case, several times each, and record the full trace of steps and tool calls.
  6. Score the runs with exact checks where possible, human review where judgment is needed, and a model acting as judge for the rest.
  7. Read the failures, group them by cause, fix the biggest group and run again.
  8. Keep the set and run it after every change, and add each new production failure as a new case.

The first set can be small. Thirty well-chosen cases teach more than three hundred copied ones. Grow it as you learn what goes wrong.

What should you measure?

A single score hides too much. Look at several dimensions and decide which ones are hard requirements for your use.

Common evaluation dimensions
DimensionQuestion it answersHow it is often checked
Task successDid it reach the right outcome?Exact check against the expected result, or human review
Tool useDid it call the right tools with valid inputs?Inspect the recorded steps
FaithfulnessDid it stick to its sources and not invent facts?Compare with source passages
Safety and limitsDid it stay within permissions and refuse what it should?Adversarial test cases
EfficiencyHow many steps, how much time and spend?Logs of calls and usage
ConsistencyDoes it do the same thing on repeated runs?Run each case several times
HandoffDid it escalate to a person when it should?Cases that require approval or are out of scope

A related risk is hallucination, where an agent states something unsupported. Faithfulness checks are the main defense, especially for agents that answer from documents.

What methods are used to score an agent?

No single method fits every case. Mature teams mix them and use the cheapest one that is reliable for each check.

Scoring methods compared
MethodHow it worksStrengthWeakness
Exact or rule-based checkCompare output or tool calls to a known value or ruleFast, cheap, repeatableOnly works where there is one right answer
Human reviewA person grades runs against a rubricBest judgment on nuanceSlow and costly; graders can disagree
Model as judgeAnother model grades the output against criteriaScales to many casesCan be biased or wrong; needs checking against human grades
Trace inspectionReview the steps and tool calls, not only the end resultFinds right answers reached the wrong wayNeeds good logging
Live shadow runRun the agent on real traffic without acting on itReal inputs, no real impactTakes time and setup

If you use a model as judge, spot-check its grades against human ones regularly. A judge that drifts will quietly corrupt every score.

Offline vs. online evaluation: what is the difference?

Offline evaluation runs before release against a fixed set. It is repeatable and safe, so you use it to decide whether a change ships. Online evaluation watches the agent in production: sampling real conversations, collecting user feedback and tracking failures as they happen.

You need both. Offline tests catch regressions before users see them. Online monitoring catches the cases you did not think of, and those become new offline cases. Tooling for the online side overlaps with AI observability and LLMOps.

What does agent evaluation look like in practice?

The cases and scores change with the agent. A few realistic examples:

  • Support agent: given real past tickets, does it find the right policy, draft a correct reply, and hand off refunds above its limit to a person?
  • Invoice agent: given a mix of clean and messy invoices, does it extract the right fields, flag duplicates and never post without approval?
  • Research agent: given a question with known sources, does it cite them, avoid unsupported claims and say when evidence is thin?
  • Coding agent: given an issue and a test suite, does it open a pull request that passes tests without touching files outside its scope?
  • Scheduling agent: given conflicting requests, does it ask a clarifying question instead of guessing?

In each case the test checks both the result and the path. An agent that gets the right answer by calling a tool it was not allowed to use has failed.

How do you build a good eval set?

Start with the work the agent will really do. Pull a sample of past tickets, emails, forms or documents, and ask the people who handle them today to mark the right outcome. Their answers become your expected results, and the disagreements between them show where the rules are unclear.

Then split the set into groups so you can see where it fails: common tasks, rare tasks, ambiguous inputs, out-of-scope requests and hostile inputs. Keep a small held-out group that you never use while tuning, and check it before release. When a production failure appears, add it to the set the same day so it cannot return unnoticed.

Write a short rubric for any case that needs judgment. A good rubric lists what a correct answer includes, what makes it unacceptable and how to treat partial credit. Two reviewers using the same rubric should land on nearly the same grade.

What mistakes make agent evaluation misleading?

  • Testing on the same examples used to tune the prompt, so the score only shows memorization of those cases.
  • Running each case once. Because outputs vary, one pass can hide a failure that happens often.
  • Checking only the final answer and never the steps, which lets unsafe paths through.
  • Using only easy cases. If nothing in the set is awkward, a good score means little.
  • Trusting a model judge without checking it against human grades.
  • Treating the set as finished. Production teaches you new cases every week.
  • Skipping the evaluation after small changes. A one-line prompt edit can change behavior across many tasks.

Evaluation also pairs with adversarial testing. AI red teaming looks for ways to break the agent on purpose, and the best attacks become permanent test cases.

What tools are used for agent evaluation?

You can start with a spreadsheet of cases and a script. As the set grows, teams move to dedicated tooling. Common options include LangSmith, Langfuse, Braintrust, Arize Phoenix, and the open-source frameworks Promptfoo, DeepEval and Ragas for retrieval-based systems. Cloud platforms such as AWS Bedrock, Azure AI Foundry and Google Vertex AI include evaluation features of their own.

Choose on fit rather than reputation: can it store traces, replay cases, support human review, run in your CI pipeline and keep data inside your boundary? The playbook on AI agent evaluation walks through those decisions.

What does evaluation mean for cost and timeline?

Evaluation takes real effort: collecting cases, writing rubrics, building the harness and reviewing failures. It also costs money to run, since every case calls the model and its tools several times. That cost is small next to shipping an agent that fails in front of customers.

Plan for it from the first sprint, not as a final step. Agents with more tools, longer workflows and higher risk need larger sets and more human review. For current model prices, see the LLM pricing page, and for typical project ranges see AI agent costs. The free estimate scopes the build including testing.

How does StackLift approach agent evaluation?

Every engagement starts with structured discovery and architect-reviewed scope, which is where the success criteria for an agent are written down. Quality, security and control are part of the governed delivery pipeline, with CI/CD, observability and rollback in the plan from sprint one.

Agents that StackLift uses to build work operate on feature branches and never push to the default branch or touch production secrets, and engineers own the decisions. The same discipline applies when we build agents for clients: defined permissions, tested behavior and a human approval step where the risk calls for it. See AI agent development for the service.

Questions people ask

How is agent evaluation different from testing normal software?

Normal tests expect one exact output. An agent can take different valid paths, so you score outcomes and behavior across many cases and repeated runs. You also check the steps it took, not just the answer.

How many test cases do I need?

Start with a small, well-chosen set that covers common tasks, awkward cases and out-of-scope requests. Grow it as production teaches you new failures. Quality and coverage matter more than raw count.

Can another AI model grade my agent?

Yes, a model can act as a judge, and it scales well. Check its grades against human ones on a sample, because judges can be biased or inconsistent. Use exact checks wherever a right answer exists.

When should I run evaluations?

After every change to the prompt, tools, data or model, and before every release. Many teams run the set automatically in their CI pipeline, then watch production for new failure patterns.

What is the difference between evaluation and monitoring?

Evaluation tests against a prepared set, usually before release. Monitoring watches live traffic after release. You need both: monitoring finds new problems, and evaluation keeps them from coming back.

Does a good evaluation score mean the agent is safe?

No. A score covers only the cases in your set. Safety also needs permission limits, human approval for risky actions, security testing and ongoing monitoring.

Who should own agent evaluation?

The team that builds the agent writes the harness, and the business owner defines what good looks like. Risk or compliance reviewers should see the results before launch.

Do I need evaluation for a simple chatbot?

Yes, though it can be lighter. Any assistant that answers customers needs a test set of real questions, checks for unsupported claims and a way to see whether changes made it worse.

Sources

Sources and further reading

The standards, rules, research and official documentation this page draws on.