AI Red Teaming Explained: What a Red Team Tests, and What It Leaves Behind

Most AI red teaming engagements are judged by the wrong artefact. A week of work produces a slide deck, the deck says the assistant can be talked into writing a phishing email, and nobody can say afterwards whether the behaviour was reproducible, how bad it was, or which control would have stopped it. The discipline has a real output, and it is not a deck: it is a set of findings that survive re-running, each one tied to a boundary that was crossed and to the evidence that crossed it. That is also the shape of the record the AI incident archive keeps — 400 entries covering December 2024 to October 2026, drawn from 743 distinct sources, and a verdict on each one about whether anyone was actually harmed.

The confusion is understandable, because "red teaming" is used for two jobs that share vocabulary and little else. This is a guide to both, to the evidence each should produce, and to the four places where AI red teaming genuinely departs from the network version most security teams already know how to run.

The framework question: who owns the test

Before scope, there is the question of who is running the exercise and against what. Microsoft publishes one of the more complete public positions on this, and it is worth reading as a map rather than as a product page. Its AI Red Team hub separates the concept of AI red teaming from the how-to, and then files the practice in an order that is itself an argument: overview, then a guide for building an LLM red team, then the reference material — a bug bar for triaging attacks on machine-learning systems, a taxonomy for machine-learning failures, a shared responsibility model, and developer threat-modelling guidance.

Read the sequence and the methodology falls out. A taxonomy decides what counts as a failure. A bug bar decides which of those failures is worth fixing. A shared responsibility model decides who is on the hook for it once it is found. Threat modelling decides where to look next. Without those four, a red team is generating anecdotes with a severity column bolted on, and the severity column is whatever the tester felt that afternoon.

Microsoft's open automation framework, PyRIT, is filed on the same shelf, and that placement is the useful part. An orchestration kit is not a methodology; it is what you point at a target once the taxonomy and the bug bar exist. Teams that begin with the framework tend to end with a large corpus of prompts and no way to rank the results.

Two red teams, two different questions

The single most useful distinction in this field is the one almost nobody writes down: a red team testing a model and a red team testing a system are answering different questions, with different units under test, different outputs, and different owners. Mixing them is how a jailbreak gets reported as a security finding, and how a genuine privilege-escalation bug gets filed as a model-quality complaint.

Model-level safety evaluation is a measurement problem. You are asking whether a model will produce a category of content, and the honest way to answer is a sample large enough to say something about a rate. The unit under test is the model behind a thin API, the method is adversarial prompting — jailbreaks, persona attacks, multi-turn pressure — and what comes out is a transcript plus a harm judgement. When the weights change or the vendor's safety layer shifts, the whole result is stale. That is why this kind of test is usually run by the model builder or by an independent evaluator, and why its findings rarely map onto a specific fix in your application.

System-level adversary emulation is an engineering problem. You are asking whether an attacker can cause the deployed thing to act — read a file it should not, call a tool it should not, spend money it should not. The unit under test is the agent with its tools, retrieval and credentials attached. The methods are the familiar ones, retargeted: injection through retrieved content rather than through the user's turn, tool abuse, privilege chains assembled out of individually reasonable permissions, data paths that leak across a tenant boundary. The output is a finding with a reproduction, a crossed boundary, and an owner who can change the configuration tomorrow.

The two produce incompatible artefacts, and that incompatibility is diagnostic rather than annoying. A model-level result cannot be "fixed" by your team except by switching models. A system-level result cannot be fixed by the vendor at all. If a report does not say which of the two it is, you cannot assign any of its items.

What the attack surface becomes once the model has tools

The surface most teams still test is the prompt. It is also the least interesting one, because it is the one the model vendor has spent the most effort hardening and the one an attacker has the least leverage over. The surfaces that actually pay are the ones the application added: the tools the agent can call, the retrieval index it reads, the memory it carries between turns, and the output channel itself.

Tools are where a language model becomes an actor, and the interesting question is never "can it call the tool" but "what is the least privilege this tool could have had". A ticket-lookup tool that accepts a free-text query can usually be talked into accepting a different customer's identifier. Retrieval is a content channel into the model's context, which means anyone who can write to the corpus — a shared drive, a support macro, a web page the fetcher reaches — is writing to the prompt. Memory turns a single turn into persistence: an instruction planted in turn two is still there in turn forty. Output is where the model's text lands somewhere that interprets it, and rendering, markdown, link previews and shell-adjacent pipelines all interpret text.

An engagement that only tests the prompt will report a clean result and be right about the wrong thing.

Why a transcript is not a finding

A finding needs three properties that a screenshot of a chat window does not have. The first is reproducibility: run it again from a clean state and it happens again. Models are non-deterministic, so reproducibility is a rate rather than a boolean — "nine of ten attempts" is a finding, "it happened once" is a rumour. The second is a crossed boundary: name the thing that should not have happened in the application's own terms, not the model's. Not "the model was rude", but "the agent read a document belonging to tenant B". The third is a severity that traces to impact rather than to the tester's intuition, which is what a bug bar is for.

Everything else — the transcript, the prompt, the model's reasoning — is evidence supporting those three. It is not the finding itself, and this is why a corpus of two thousand jailbreak prompts is worth less than five reproducible system findings with named controls attached.

Four places AI red teaming diverges from network red teaming

Teams arriving from network or application security bring good instincts and four wrong assumptions.

• There is no perimeter to hold. A network engagement has a target list; an agent engagement has a context window assembled at request time from sources you may not have inventoried. Scope has to be expressed as data sources and tool permissions, not hostnames.

• The target is non-deterministic. The same input can succeed on Tuesday and fail on Wednesday. This breaks the "one clean reproduction" evidence standard and forces you to report rates.

• The exploit is language, and language has no patch. You can fix a missing authorization check. You cannot patch "the model can be persuaded". The mitigation for a language-level weakness is a control somewhere else — a filter, a scope limit, a permission boundary — which is why findings need to name the control, not just the behaviour.

• The vendor is inside the loop. Your target changes when someone else ships a version. A model-level result is only valid for the build you tested, and if you did not record the version and the date, you do not have a result.

How to scope a first engagement

The engagements that go badly are almost always scoped as "test the AI". A scope that survives contact with a schedule looks more like this: one agent, in a staging environment, with production-shaped data and production-shaped permissions; a list of the five boundaries that matter (tenant isolation, credential access, spend, data egress, tool scope); rules of engagement that name what is out of bounds; stop conditions that say who can halt the run; and a defined evidence format agreed before the first attempt, so that the report is assembled from artefacts rather than reconstructed from memory.

Authorization deserves its own sentence, because it is the one item teams skip when the target is their own product and the tools are someone else's. Testing a model you do not operate, on infrastructure you do not own, with prompts designed to defeat a safety layer, is a different activity from testing your own deployment — and the written scope is what separates the two. The verification gate our own cyber tier runs on exists for the same reason: authorized security work at the model level needs a named, accountable party on both sides.

What the incident record adds

A red team run tells you what your target does. It cannot tell you what happens in the field, and the field is where the argument for a control actually gets made. That is the gap an incident ledger fills — and the discipline it imposes is the useful part: a record without at least one source does not exist, and a record without a harm verdict is not an incident.

Read the two artefacts together and the loop closes. The red team produces a reproducible finding and names the control that would have blocked it. The record tells you whether anyone in the field has already hit that boundary, what the harm was, and whether the control held anywhere else. A finding with no field precedent is a hypothesis worth fixing; a field incident with no reproducible finding is a bug you cannot yet close. The teams that get this right stop treating "we red-teamed the model" as a deliverable and start treating it as one input to a ledger of their own.

Which leaves the honest closing note. AI red teaming does not produce a security guarantee, and any engagement sold as one is selling the wrong thing. It produces a set of boundaries you now know are crossed, with evidence you can re-run, ordered by what happens when they are. That is enough to make decisions with — and considerably more than a deck.