StackLift
All terms

Glossary

AI red teaming

Deliberately attacking your own AI system to find failures, unsafe behavior and security holes before others do.

AI red teaming is the practice of deliberately probing an AI system, as an attacker or a misuser would, to find security flaws, unsafe outputs and unintended behavior before release. Testers try prompt injection, data extraction, misuse of tools and policy bypasses, then report what worked so the team can fix it.

The term comes from security, where a red team plays the attacker. For AI it covers more than security: bias, harmful content, privacy leaks and reliability failures are all in scope, alongside attacks like [prompt injection](/glossary/prompt-injection).

Red teaming can be done by people, by automated tools that generate attacks, or both. Findings feed back into [guardrails](/glossary/ai-guardrails) and the test set used for [agent evaluation](/glossary/agent-evaluation).

It is not a one-time certificate. Models, prompts, tools and attacks change, so the exercise is repeated.

Start in 60 seconds

Put this to work on your project.

Get an AI-assisted, architect-reviewed scope, cost range, and timeline. Free, no login.

The complete guide

AI red teaming: the complete guide

Updated November 18, 2026 ยท Published by , Founder, StackLift AI. Written with AI assistance; how we make content.

Short answer

AI red teaming is structured adversarial testing of an AI system. People, and sometimes automated tools, try to make it leak data, ignore rules, take harmful actions or produce unsafe output, so you find weaknesses before real users or attackers do. Findings become fixes and permanent regression tests.

What is AI red teaming?

Red teaming is the practice of playing the attacker. A team deliberately tries to break your AI system, using the same tricks a hostile user, a careless user or a malicious document might use. The goal is to find the failures while you can still fix them cheaply.

The idea comes from security and military exercises. In AI it covers more than classic security holes. It also covers safety and quality failures: harmful content, biased answers, made-up facts, private data in the output and agents that take actions they should not.

Red teaming is not the same as ordinary testing. Ordinary tests check that the system does what it should. Red teaming checks what happens when someone tries to make it do what it should not.

Why do you need red teaming?

Language models are flexible, and that flexibility is the weakness. The same system that follows a clever instruction from you will follow a clever instruction from an attacker, or from text hidden in a web page it reads. You cannot list every attack in advance, so you need people who think creatively about misuse.

  • Systems with tools can act, so a successful attack can change records, send messages or move money, not just say something wrong.
  • Systems connected to private data can leak it through clever questions or through prompt injection.
  • Public-facing assistants represent your brand, and one screenshot of a bad answer travels fast.
  • Regulations and customers increasingly ask how an AI system was tested for misuse and what controls exist.

Red teaming shows whether your guardrails hold up under pressure. A guardrail you have never attacked is an assumption.

What does a red team try to do?

Common attack goals
GoalWhat it looks likeTypical defense
Prompt injectionHidden instructions in a document, email or web page that the model obeysTreat retrieved content as data, limit tool permissions, require approval
JailbreakingWording that talks the model out of its safety rulesLayered input and output checks, tested instructions
Data leakageGetting the system to reveal system prompts, other users' data or private documentsPermission checks on retrieval, output filters, no secrets in prompts
Harmful or off-policy outputContent the business never wants producedOutput checks, topic limits, human review for sensitive flows
Excessive agencyMaking an agent use a tool it should not, or use it too broadlyLeast-privilege credentials, schema validation, approvals
Resource abuseLong loops or huge requests that run up costStep, time and spend limits
Bias and unfair outputDifferent quality or tone for different groups of peopleTest sets across groups, review, clear escalation

Industry lists of common risks for LLM applications, such as the OWASP list for LLM applications, are a useful checklist for what to cover. Use them as a starting point rather than a complete plan.

How does an AI red team exercise work?

  1. Define scope: which system, which version, which tools and data, and what is out of bounds.
  2. Set the harm list: what outcomes would matter most to the business, users and regulators.
  3. Map the attack surface: every input, document source, tool, permission and output channel.
  4. Attack manually: testers try creative prompts, poisoned documents and misuse of tools.
  5. Attack at scale: automated tools generate and mutate many attack prompts and score the results.
  6. Record each finding with steps to reproduce, severity and the likely cause.
  7. Fix the cause, preferring controls enforced in code over extra prompt text.
  8. Add each finding to a regression suite and retest after every change.

The output is not a report that sits in a folder. It is a set of fixes and a growing test set that protects the system as it changes.

Manual or automated red teaming: which do you need?

Both. Automated tools find breadth. People find creativity and context. Humans spot attacks that depend on understanding your business, such as a request that is harmless in general but harmful in your workflow.

Manual versus automated testing
ApproachStrengthLimitBest use
Manual testingCreative, understands context, finds new attack patternsSlow, depends on the tester's skillBefore launch and after major changes
Automated attack toolsFast, repeatable, broad coverage of known patternsMisses novel and business-specific attacksEvery release, as a regression gate
Model-driven attackersCan adapt attacks over many turnsNeeds review, can produce noisy resultsExploring multi-step abuse
Bug bounty or external reviewFresh eyes and diverse thinkingNeeds triage and clear rulesMature, public-facing systems

Open-source tools exist for generating and scoring attacks, for example Garak, PyRIT and promptfoo. Choose by the kinds of systems you test and the checks you need.

Why do agents need extra red teaming?

A chatbot can only say things. An agent can do things, so the damage from a successful attack is larger. Tool calling, memory and long-running plans open new paths: an attacker may plant an instruction in one step that triggers harm many steps later.

  • Test every tool for what happens when the model passes unexpected arguments.
  • Test what the agent does when a retrieved document contains instructions.
  • Test whether one user can influence what the agent remembers about another. See agent memory.
  • Test handoffs in multi-agent systems, where one agent's output becomes another's input.
  • Test that approvals cannot be skipped or implied by the model's own wording.

The strongest fixes are structural: narrow permissions, validated inputs and human approval for actions that are hard to undo.

What does a real exercise find?

These are typical categories of findings, not results from any named engagement.

  • A support assistant reveals part of its system prompt when asked to repeat its instructions in a different language.
  • A document summarizer follows an instruction hidden in white text inside an uploaded file.
  • A sales agent can be talked into offering a discount outside policy because the limit lived only in the prompt.
  • An internal search tool returns snippets from documents the asking employee should not see, because permissions were checked after retrieval rather than before.
  • A coding agent writes secrets into a log file when asked to debug an authentication problem.

Notice that most fixes are not about the model. They move a rule out of the prompt and into code, or reduce what a tool is allowed to do.

How do you run a first exercise on a small system?

  1. Write down the worst five things the system could do with its current access.
  2. Pick two or three people who did not build it, including one who knows the business process.
  3. Give them a short list of attack ideas: ignore-your-rules prompts, hidden instructions in documents, requests for other users' data, odd tool arguments.
  4. Let them try for a fixed time and record every success with exact steps.
  5. Rank the findings by what harm they could cause, not by how clever they were.
  6. Fix the top items with controls in code, then repeat the same attacks.
  7. Save every attack as a test that runs on each release.

A day of focused effort on a narrow system often reveals more than weeks of ordinary testing, because it changes the question from "does it work?" to "how can it fail?".

How do you measure red teaming results?

Avoid a single score that claims the system is safe. A clean run shows only that these attacks did not work today. Track the shape of the work instead.

  • Findings by severity and by category, and how many are fixed.
  • How often known attacks succeed after each release.
  • How quickly new findings are fixed and retested.
  • Coverage: which tools, data sources and flows have been attacked.
  • Over-blocking: how often legitimate requests are refused after a fix.

Pair this with agent evaluation for quality, and with AI observability so production traffic reveals attacks you did not think of.

When should you red team, and who should do it?

Start before launch, repeat after major changes such as a new model, new tools or new data sources, and keep a lighter automated pass running on every release. The first exercise is the most revealing; later ones find regressions and new surface area.

Testers should be separate from the builders where possible, since builders tend to test the paths they expect. Mix security engineers, people who know the business process and, for sensitive areas, domain experts. Give them written rules about scope and about handling any sensitive data they find.

What mistakes should you avoid?

  • Testing only the model in isolation and not the full system with its tools and data.
  • Treating a single exercise as permanent. Models, prompts and tools change.
  • Fixing findings by adding more prompt text instead of an enforced control.
  • Not saving findings as regression tests.
  • Running tests against production systems with real customer data and no rules of engagement.
  • Keeping results private from the team that has to fix them.

What does red teaming mean for cost and timeline?

Effort depends on the system's reach: the number of tools, the sensitivity of the data, the size of the public audience and how much regulatory attention the use case draws. A narrow internal assistant needs a smaller exercise than a public agent that can change customer accounts.

Plan time for fixes and retesting as well as for attacking, because the value comes from closing the findings. The free estimate scopes testing alongside the build, and the security review checklist lists what to cover.

How StackLift approaches it

Security review is part of the plan from sprint one, and adversarial cases are scoped as deliverables next to the controls they test. Agents work on feature branches and never touch production secrets, and engineers own the decisions.

See AI agent development, the AI governance policy checklist and our security page.

Questions people ask

Is red teaming the same as penetration testing?

They are related. Penetration testing looks for technical holes in software and networks. AI red teaming adds model-specific attacks, such as prompt injection and jailbreaks, and also checks safety and quality failures.

Can a small company do AI red teaming?

Yes. Start with a written list of the worst outcomes, a few hours of creative attempts by people who did not build the system, and an automated test pass. Scale up as the system's reach grows.

Does red teaming make an AI system safe?

No. It finds weaknesses so you can fix them, and it cannot prove none remain. It is one part of a program that also includes guardrails, monitoring and human approvals.

How often should I red team my AI system?

Before launch, after major changes such as a new model or tool, and on a regular schedule. Run an automated regression pass on every release.

What is the difference between red teaming and evaluation?

Evaluation measures how well the system does its intended job on expected inputs. Red teaming measures how it behaves under hostile or unusual inputs.

Who owns the findings?

The team that owns the system fixes them, with a named person tracking severity and retests. Security and the business owner should see the open items.

Do I need red teaming if I only use a hosted model through an API?

Yes. The provider tests its model, but your prompts, data, tools and business rules are yours, and attacks usually target those.

Sources

Sources and further reading

The standards, rules, research and official documentation this page draws on.