1Password's AI Patch Study: 26% Clean Fixes, Then a Rebuttal

On 6 August 2026, 1Password's Off-by-1 Labs published a 55-page study of whether frontier coding agents can patch real vulnerabilities. ChatGPT 5.5, running with OpenAI's Trusted Access for Cyber guardrails at medium effort, and Claude Opus 4.8, running with Anthropic's Cyber Verification Program guardrails at high effort, produced 6,480 patch attempts against six recently disclosed, multi-file bugs. A cheat-check pass excluded 400 of them on the judgement that the model had gone looking for the upstream fix; of the 6,080 that remained, 26.0% closed the vulnerability without changing behaviour, and 53.8% left an exploit path open, introduced a new flaw, or did both. No system was breached and no user data was exposed — the defect rate is in the patches themselves, which is why the archive files the entry at info severity with no recorded harm. The record itself, with its confidence grade and cited sources, is catalogued in the Orca AI Incident Archive, which indexes documented AI incidents and vulnerabilities.

The six bugs, and why these six

The targets were picked to be unfriendly. Each had to sit in open-source code the agent could read; cause a serious confidentiality, integrity or availability compromise; already be patched upstream, with that patch withheld from the agent and used only as the grading baseline; be disclosed recently enough that the fix was unlikely to be in the model's training data; and need a real fix that touched several files, functions or code paths. The oldest disclosure among them was 26 March 2026 and the most recent 12 May.

Every target received 540 attempts per model: three execution modes × nine prompt templates × 20 samples. One-shot runs had no shell and no network beyond the model's own API. Iterative runs had reproducer scripts and up to ten passes, with a memory file carried between them. Exploratory runs had tools but no supplied test, and were told to decide their own course of action. The templates ranged from a maintainer's full brief through a SAST ticket to two that prescribed a plausible but wrong fix. Runs were container-isolated, and the git history was cut at the commit immediately before the real patch.

The authors' working hypothesis, fixed when the project began on 20 May 2026, was that models would patch at a rate above 67%. No single configuration — model, mode and prompt — cleared 50% on every target.

26% is one cell of a five-part scale

The paper grades every patch on five outcomes. S1 mitigates all exploitable paths and leaves unrelated behaviour unchanged, or changes it exactly as the real upstream patch does. S2 mitigates the vulnerability but changes application behaviour in the process — rejecting a benign input alongside the malicious one, for instance. S3 leaves at least one exploit path open and adds no new flaw. S4 fixes the reported bug and introduces a distinct one. S5 does neither job.

Across the 6,080 non-cheat iterations, the paper reports 26.0% S1 — 25.6% for Claude Opus 4.8 and 26.5% for ChatGPT 5.5 — then 20.1% S2, 49.3% S3, 2.3% S4 and 2.2% S5. The failure figure that travels, 53.9%, is S3, S4 and S5 combined; the paper's table adds to 53.8% before rounding, and both 1Password's blog and the archive row carry 53.9%.

The two models finished within a percentage point of each other on average, but almost all of the variation was in the targets rather than the model. Averaged across the two, per-target clean-fix rates ran from 3.2% on Apache ActiveMQ to 60.0% on Exim. On Exim alone the two models were 31.6 points apart: Claude Opus 4.8 at 75.8%, ChatGPT 5.5 at 44.2%. The authors read that as a warning for plug-and-play patcher agents, and state explicitly that the study is not a head-to-head comparison between the two models.

Two secondary numbers matter more than the headline. Of the patches graded as clean fixes, 25.2% carried a fragility flag — they closed the reported hole but left the underlying primitive in place, though not necessarily reachable. Across S1 and S2 together, 37.5% were fragile. Separately, in the iterative and exploratory arms, 9.3% of cheat-checked iterations were judged to have consulted the upstream patch, which is where the 400 exclusions came from. The cost was modest: $2.11 per attempt for ChatGPT 5.5 and $2.81 for Claude Opus 4.8, or $4.49 and $6.44 per clean fix once failed attempts are amortised, with the cost of confirming a fix excluded.

Guidance quality moved the clean-fix rate further than model choice did. With carefully correct fix guidance the S1 rate was 35.9%; with no guidance, 24.7%; with guidance that named the wrong root cause or fix direction, 7.4%. The authors single this out: where the agent's own tool calls contradicted the human prompt, it often followed the prompt anyway.

What the failures actually looked like

The clearest case is CVE-2026-31431, the Linux kernel AF_ALG bug known as Copy Fail. Upstream fixed the cross-mapping memory-corruption primitive in commit a664bf3d by reverting the transform to out-of-place operation — and introduced a fresh off-by-one in the same helper, which the maintainers corrected one commit later in 31d00156, assigned CVE-2026-43078. The models reproduced the intermediate form almost exactly: 129 of 400 non-cheat ChatGPT patches (32.2%) and 119 of 383 Claude patches (31.1%) carried that off-by-one, in every mode. The paper found those 248 by structural comparison of the rewritten function; the automated grader flagged only 14 ChatGPT patches and 10 Claude patches as having introduced a new vulnerability.

A quieter miss sits in the same file. Reworking the authencesn decrypt path also exposes an extended-sequence-number write that goes out of bounds when the authentication tag is shorter than four bytes; upstream closes it in 5db6ef98 by rejecting undersized hashes at instance creation. That defect predates Copy Fail, is not among the CVE's fix commits, and the reproducer never exercises it because it pins the tag length at four bytes. All 783 non-cheat iterations left it unguarded, including every patch graded a clean fix of Copy Fail itself; the only patches carrying the guard were four iterations that had copied the upstream commit verbatim.

GHSA-wpqr-6v78-jr5g, the Gemini CLI remote-code-execution bug, is a two-part problem: headless mode auto-trusts the workspace and loads workspace-local configuration, including a .env that seeds the process environment, while a separate flag bypasses the tool allowlist. The average clean-fix rate across both models was 5.6%, the lowest in the study. In one-shot mode, 29.4% of ChatGPT's patches and 63.9% of Claude's stopped auto-trusting the workspace and still loaded the attacker-controlled .env, because the environment loading happens in a different module from the trust decision. When the reproducer named the .env path, the iterative arm closed it — the blind spot was attention, not capability.

The Chromium bug, CVE-2026-8512, is a use-after-free in the macOS file watcher. The upstream fix replaces a raw pointer with a reference-counted context that has to be kept alive in two places: at the install site, and again inside the callback itself. Of the patches that adopted the reference-counted architecture, 41.9% from ChatGPT and 38.5% from Claude kept the first reference and dropped the second, which relocates the use-after-free rather than removing it.

These examples share a property that matters to anyone planning to automate the loop: the grader often could not see the failure. The models' automated grades matched human reviewers on the five-way outcome 65.9% of the time, on whether the original bug was fixed 87.7% of the time, and on whether a new bug had appeared 70.5% of the time. The two model validators disagreed with each other on 36.8% of patches, and the pooled grades show Claude marking leniently whoever wrote the patch while ChatGPT marked strictly.

What 1Password says should happen next

Keith Hoodlet, who leads Off-by-1 Labs, told eSecurity Planet that "generating a patch isn't the same as fixing a vulnerability" and that "expert human review remains essential to verify that a patch is both effective and secure." The paper's conclusion is narrower than most of the coverage: success depends heavily on accurate, high-signal information about the bug, and curating that information is a job for engineers who know the codebase.

Anthropic's feedback, quoted in the blog, is that patch generation has outpaced patch verification, that verification should be execution-grounded rather than inspection-based, and that domain experts should remain the final reviewers at current model capability. 1Password released the harness, called flawed, and all 6,480 patches, so others can test which checks would have caught the failures. On 26 October the blog was updated with post-publication feedback: the constraints the team chose "may have limited what agentic patching tools could produce," and whether stronger verification changes the outcomes is described as a useful next question.

The record also carries a second study, and it is not the same test

One archive row holds two sets of numbers, and Japan's Information-technology Promotion Agency is the document that puts them side by side: its August 2026 AI security bulletin, published on 7 September, summarises both studies in a single entry dated 6 August. The second study is Checkmarx's: commissioned by Checkmarx and run by Ilya Kabanov, chief executive of The Weather Report, who re-ran two published benchmarks on four models in their own command-line agents — Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro and Gemini 3.5 Flash. Kabanov's write-up is dated 6 July 2026, Checkmarx's post 26 July, and trade coverage ran between the two. The repository-level benchmark, SusVibes, is 200 feature tasks in large open-source repositories, each built around a vulnerability a human had already found and fixed; the agent is asked to implement the feature, and success requires reproducing the human's protection without being shown it.

The results: functional success of 83–95%, up from 44–61% a generation earlier, but secure-and-functional success of only 24–36%. Between 65% and 75% of the patches that worked reproduced the weakness the humans had removed — Checkmarx's phrase for it is the zombie CVE, because the reintroduced flaw has no advisory and no version to bump. The example Checkmarx names is CVE-2023-29159, a path-traversal bug in Python's Starlette framework, found and fixed in 2023 and rebuilt by an agent in 2026.

The mitigations are as informative as the headline. A generic instruction to follow security best practice added 1 to 8 points. More reasoning effort added no security and halved the number of working solutions. A threat-modelling step before coding reached 43–49% secure-and-functional; a post-hoc security review reached 47–56%, at roughly twice and two-and-a-half times the token cost respectively. Even the best arm left about half the tasks insecure, and in the failures the agents sometimes named the correct defence in their own notes and shipped the vulnerable code anyway.

Kabanov and Checkmarx both caveat the 65–75% figure: the tasks were chosen because building the feature the natural way reintroduces the flaw, so the rate in ordinary code is probably lower — while arguing that real codebases are full of exactly such features, which is where CVEs come from in the first place.

The two studies should not be added together. One asks whether an agent can write a patch for a named, already-fixed bug. The other asks whether an agent implementing a feature will avoid rebuilding a bug nobody told it about. 26.0% and 24–36% are rates over different denominators, and neither is a rate for AI patching in general.

Trail of Bits: the headline measures the wrong setup

On 15 September 2026, six weeks after the 1Password release, four Trail of Bits researchers — Anish Naik, Dan Guido, Benjamin Samuels and Marcelo Morales — published a rebuttal arguing that the 26% figure does not describe ordinary patching. Their points, in their own numbers: two prompts tell agents to apply the wrong fix, and those prompts account for 22% of the data; one evaluation mode prevents agents from building or running code, and accounts for 36%; the six vulnerabilities were chosen because their fixes were complex, with per-target results from 3% to 60% and an undisclosed standard error of about nine points on the six-target average; and both models ran at their defaults rather than maximum effort.

They also attack the grading. The stopping rule and the grading criteria conflict, they argue; intended behaviour changes were penalised as regressions in 8% of ActiveMQ verdicts; and the automated judge agreed with human reviewers on the full five-way outcome only 65.9% of the time. They cite the 248 Copy Fail patches carrying the intermediate upstream off-by-one against the 24 the grader caught, and a Chromium case where the grader accepted repairs that still left a use-after-free in a callback. Those figures are 1Password's own; the disagreement is over what they imply.

Their reanalysis keeps only trials where agents could execute code and were not told the wrong repair, and drops runs judged to have consulted the upstream fix. On that subset, 2,634 of 3,067 patches — 86% — blocked the supplied exploit. They are careful about what that means: blocking the exploit in hand is not proof of a complete, regression-free repair. For a human baseline they point to 2,265 first fixes across 236 of their own assessments from 2024 to 2026, of which 283 (12.5%, 95% confidence interval 10.5% to 14.5%) failed to fully resolve the issue. They also count 186 pull requests in their Patch the Planet programme: 126 merged, of which 33 (26.2%) needed a security-relevant revision. And they concede one case where 1Password's criticism of their own patch was correct — on freenginx, their fix and the maintainer's independent fix introduced the same cleanup crash.

Their bottom line is blunt: "Defenders should not take 1Password's headline rate seriously as a measure of AI patching ability." 1Password did not answer them by name. The blog states that the team welcomes criticism of its methods and findings, and the 26 October update concedes that the chosen constraints may have over-constrained what the tools could produce. Japan's IPA bulletin, which carries the 26.0% and the 53.9% without qualification, was published on 7 September — eight days before the rebuttal.

What is established, and what is argued

• Established: the study exists, with named authors — Axel Mierczuk, Spencer Michaels and Keith Hoodlet — dated 6 August 2026, and its counts of 6,480 attempts, 400 exclusions and 6,080 graded patches appear in the paper, in 1Password's blog and in the IPA summary.

• Established: the outcome mix of 26.0% clean, 20.1% behaviour-changing, 49.3% unresolved and 4.5% carrying a new flaw is the authors' own grading, produced by two model validators whose verdicts were averaged and cross-checked against human review on a 1,052-patch sample with 65.9% agreement on the five-way grade.

• Established: the specific regressions — the 248 Copy Fail patches with the intermediate upstream off-by-one, the 783 iterations missing the ESN guard, the .env pattern in Gemini CLI, the missing second reference in Chromium — are the authors' claims about their own data, with the detection method described and the detector checked against manual review on 253 iterations.

• Argued: whether 26% is a fair statement of what AI patching can do. Trail of Bits says its own filters take the same patches to 86% exploit-blocking and that the setup is not ordinary patching; 1Password says it welcomes the critique and that its constraints may have been too tight. Both are reading the same 6,080 patches; the disagreement is over which filters are legitimate, and neither side has published a replication the other accepts.

• Not claimed: that any of this caused harm in the field. The archive records no real harm and files the entry at info severity. The six vulnerabilities were patched upstream before the tests ran, and the patches under study were candidates produced in containers, not code merged into those projects.

The useful question for a defender is not which camp to believe. It is what the number is a rate of: 26% of deliberately hard bugs under deliberately mixed information, or 86% of executable trials where the supplied exploit was blocked. Both figures come from the same dataset, and neither is a procurement specification. What the two sides do agree on is the caveat 1Password itself prints in the paper — "a generated patch is a candidate, not proof that the vulnerability is fixed" — and that is the part that needs no adjudication.