LLM Jailbreaks Explained: What a Jailbreak Changes, and Why It Is Not Abliteration

In July 2023, three researchers ran 32 harmful prompts against GPT-4 and Claude 1.3 through 28 different attacks. The strongest single technique — three attack families stacked on top of each other — got through on 94 per cent of the set against GPT-4 and 81 per cent against Claude 1.3. What closed the set was not a better attack but a choice: an adversary allowed to pick whichever of the 28 worked best for each prompt got all 32 through on both models, which the paper reports as a clean 1.00. The uncomfortable reading is that a jailbreak is not a defect that disappears when the model gets smarter or better behaved. It is a property of the prompt, and the prompt belongs to whoever is sending it. What happens after the refusal stops — the tool call, the file read, the output that lands somewhere that acts on it — is the part the AI incident archive was built to record.

That framing is worth being precise about, because "jailbreak" is used for several different things and the differences decide what any mitigation can even do. This piece covers what a jailbreak edits, what the measured numbers actually say, why there is no per-model answer to "is it jailbreakable", and where the technique stops being a model-safety question and becomes a system-security one.

A jailbreak is an edit to the prompt, not to the model

The model stays exactly as shipped. Nothing is retrained, no weights are touched, no fine-tune is applied. The attacker changes the request so that complying becomes the locally correct continuation, and the safety behaviour that was trained into the model loses to a competing instruction it was also trained to follow.

That single sentence rules out a large category of supposed fixes. You cannot patch it at the weights — there is nothing wrong with the weights. You cannot enumerate it, because the space of prompts is open. And you cannot check it once, because the technique that fails today is not the one that will be tried tomorrow. What you can do is measure a rate, on a fixed prompt set, against a fixed model version, and then watch that rate move when any of those three change.

What the 32-prompt experiment actually measured

The study is Jailbroken: How Does LLM Safety Training Fail? by Alexander Wei, Nika Haghtalab and Jacob Steinhardt, posted to arXiv in July 2023 as 2307.02483. Its test set was 32 harmful prompts — half drawn from a published GPT-4 evaluation, half adapted from published red-teaming work on Claude — and its outcome measure is blunt: a case is scored as a success when the reply is both harmful and on-topic for the original request. Anything that stalls, deflects or answers a different question does not count.

Two properties of the results are more useful than any individual row. The first is how unevenly attacks transfer between models. AIM role-play got 0.75 against GPT-4 and 0.00 against Claude 1.3. Few-shot JSON got 0.53 and 0.00. Meanwhile Base64 encoding was the reverse kind of uneven — 0.34 on GPT-4 and 0.38 on Claude 1.3 — and distractors landed at 0.44 and 0.47. A technique that a vendor has clearly trained against reads as a zero on that model and says nothing about the model next to it.

The second is what happens when families are combined. Prefix injection plus refusal suppression plus Base64 produced 0.56 against GPT-4 and 0.66 against Claude 1.3. Adding style injection took it to 0.69 and 0.84. Adding website-content generation and formatting constraints took it to 0.94 and 0.81. Stacking is the whole trick: each layer is something the model was trained to do, and the combination leaves no token position where refusing is the more natural continuation.

The models in that table are three years old and no longer what anyone is deploying. The numbers are not the point; the shape is. Every mechanism in the table is still live in every model since, because none of them was ever a bug in GPT-4 or Claude 1.3 — they are consequences of how instruction-following and safety training interact.

Two ways safety training fails

The paper's explanation is unusually clean, and it is the part worth carrying around: there are two failure modes, and every attack in the table is an instance of one of them.

Competing objectives are what you get when a model's capabilities and its safety goals point in different directions. Instruction-following is a capability. When the request is wrapped so that refusing would violate an instruction the model is strongly trained to obey, the model resolves the conflict in favour of the instruction. Prefix injection is the minimal version: tell the model its reply must begin with "Absolutely! Here's", and the first tokens are already a commitment that makes refusal incoherent. Refusal suppression attacks the same seam from the other side, by adding formatting rules that forbid the sentence a refusal would have to be written in.

Mismatched generalization is what you get when safety training covers a narrow input distribution that the model's general capability does not. The model learned Base64 during pretraining and can follow an encoded instruction perfectly well. Safety training, which mostly saw natural-language requests, never taught it to recognise that an encoded request is still a request. The same failure shows up with disemvowelment, ROT13, leetspeak, payload splitting and distractor tasks: each is a format the capability generalises to and the safety behaviour does not.

Once you have those two categories, the attacks stop looking like a grab-bag of tricks and start looking like two families with a shared root cause — which is also why they compose so well. A prompt that suppresses refusal and hides its intent in an encoding is exploiting both failure modes at once, and 0.94 is what that looks like.

Why there is no per-model answer to "is it jailbreakable"

The most common query shape in this space is a model name followed by the word jailbreak. It is the wrong unit of analysis, and the reason is the table above. Success depends on at least five things at once: the base model's capability, what safety training it received and on which distribution, the prompt format, the sampling settings, and how many attempts the attacker is willing to spend.

That last one deserves its own sentence. A rate measured at one attempt per prompt and a rate measured at "best of 28" are not the same measurement, and the paper's adaptive result — 1.00 on both models — is the second kind. Anyone quoting a jailbreak rate without saying which one they mean has quoted nothing usable. The realistic defensive question is not "can it be broken" but "how many attempts does it take, and does anything in my stack notice the attempts".

Two practical consequences follow. A model that resists your prompt set today can stop resisting after a version bump, a safety-layer change or a serving-side configuration change — the result is only valid for the build you tested, which is why a test without a version and a date is not a result. And the same prompt set run against two models tells you which one your stack is softer on, not which one is safe.

What a jailbreak is not: abliteration

These two get conflated constantly because both end with a model producing output it was trained not to produce. They are not the same edit, and the difference is the difference between a control you can add and a property you cannot.

• Jailbreak — a prompt-layer edit. Weights unchanged. The refusal direction is still in the model; the request has simply been arranged so that other behaviours outrank it. Revert the prompt and the behaviour reverts with it.

• Abliteration — a weight-layer edit. The refusal direction is identified — usually as a direction in activation space — and projected out of the model's weights, so that the model no longer produces refusals for that class of request at all. It is a permanent change to the artifact, and it is why an abliterated checkpoint behaves differently from the same base model with a clever prompt. The mechanics of that edit are set out in our guide to abliteration.

The operational difference matters more than the terminology. A jailbreak is a runtime event: it arrives with a request and can be counted, rate-limited, filtered and logged like any other request. An abliterated model is a supply-chain event: the control has already been removed before you receive it, and no amount of prompt-layer monitoring brings it back. If your threat model is "someone will try to persuade the model", you are in the first category. If it is "someone will ship me a model with the refusal removed", you are in the second, and the only real control is provenance — knowing which checkpoint the endpoint is actually serving.

Jailbreak is not prompt injection either

The third term that gets mixed in is prompt injection, and the distinguishing question is who the attacker is. A jailbreak comes from the user, who is trying to get content the model was trained to withhold. Prompt injection comes from content — a retrieved document, a web page, a tool result — that carries instructions the developer never wrote, and it works whether or not the user is hostile. The same model can be immune to one and wide open to the other, because they are defeated in different places: jailbreaks at the input and output of the model, injection at the trust boundary between data and instructions.

What actually reduces the rate

Nothing here is a guarantee, and any claim otherwise is a sales pitch. What follows is the set of controls that changes the numbers, ordered by how much they buy for the effort.

• Assume the prompt layer falls. Every control that depends on the model refusing is one adaptive attacker away from zero. Design as though the request will eventually get through, and put the control where the harm would happen.

• Restrict what a successful jailbreak can reach. The blast radius of a jailbroken assistant is whatever its tools, credentials and retrieval scope allow. Least privilege on tools and per-tenant scoping on retrieval reduce the value of a successful jailbreak far more reliably than any prompt filter.

• Judge the output, not the prompt. Output-side classifiers catch families you have never seen, because they look at what was actually produced rather than at how it was asked for. They are also the only layer that still works when the input format is novel — which, per the mismatched-generalization failure mode, is most of the time.

• Treat the system prompt as a configuration value, not a security boundary. It is text in the same context window as the attack. It raises the cost of an attempt; it does not stop one.

• Make the attempts visible. The one advantage a defender has over the adaptive attacker is that the attack is usually repeated. Rate limits, repetition detection and logging turn a 28-attempt search into an event you can see rather than 28 requests that each looked fine alone.

• Re-test on every model change. A version pin plus a fixed prompt set is what turns "we tested it once" into a number you can compare against. Without the pin, the comparison is meaningless.

How to measure your own target

If the goal is a number you can act on, fix the three variables the paper fixed: the prompt set, the model version, and the number of attempts. Use a prompt set that includes both failure modes — instructions that make refusing awkward, and encodings that hide the request — because a set made only of one will flatter a model that is weak to the other. Score conservatively: harmful and on-topic, not merely off-tone. And report the result as a rate against a stated number of attempts, never as a yes or no.

Where several candidate models are in play, the comparison only works if the same prompt set runs against each of them, and that is the one place the plumbing matters more than the methodology. Running one evaluation across the models we route removes the per-vendor contract and client work from the exercise; provider list prices pass through at zero markup, so the run costs what the providers charge and nothing more. What the evaluation measures is the model's behaviour, not ours — routing changes which model answers, and it does not make any of them refuse more.

Where the record picks up

A measured success rate tells you how hard the prompt layer is to push over. It does not tell you what happened when someone pushed it over in production, and that second question is what decides whether the rate matters. A jailbreak that produces a rude paragraph is a content-moderation problem. A jailbreak that produces a tool call is an incident, and the difference is entirely in what the system let the output reach.

That archive is the field record: what agents actually did, with a source for each entry and a verdict on whether anyone was harmed. Read against a jailbreak measurement, it answers the question the measurement cannot — whether the boundary that a prompt crossed is one that anything downstream depends on. If it is, the jailbreak is a security finding and the control belongs in the tool permissions. If it is not, the jailbreak is a content result, and the honest move is to file it as one rather than inflate it into a vulnerability.

The closing note is the one the evidence supports. Jailbreaking will not be solved by a better model, because the failure modes are structural: instruction-following is a capability and safety training is a narrower one, and the gap between them is where every attack in this article lives. What can be solved is the consequence — by making sure that a model which stops refusing still cannot do very much.