CyberGym Benchmark Explained: What the E2E Score Actually Measures

The CyberGym score that circulates most widely — the number in the launch post, the one that gets compared across models — is a task success rate over 131 filtered tasks, where success means an agent produced a proof-of-concept input that crashes an unpatched C/C++ build, wrote a patch that stops that crash, and did not break the project's developer-written tests. On BenchLM's compilation, MiMo-V2.6-Flash leads at 95.1% and MiMo-V2.6-Pro at 94.0%, with DeepSeek-V4.1-Flash a few points behind in the high eighties. Almost nobody quoting that table mentions three things that decide whether the numbers are comparable: the benchmark exists in more than one implementation, most of the rows are provider self-reports rather than independent runs, and the single most interesting stage is recorded but not counted. That is the same standard of attribution the AI incident archive applies to its own entries — a claim without a source is not a result — and the CyberGym corpus fails it more often than its reputation suggests.

This is a guide to what the benchmark asks an agent to do, how the task set was built, where the implementations diverge, and what a defender should and should not conclude from a high score.

The task: find, reproduce, patch — without breaking anything

CyberGym comes from the Berkeley Center for Responsible, Decentralized Intelligence, and its premise is deliberately narrow. Each task targets a memory-safety vulnerability in a widely used open-source C/C++ project. The agent must locate the bug, write a proof-of-concept input that triggers a sanitizer crash, and then patch the code so the crash no longer reproduces — while the project's own tests continue to pass. That sequence is the whole point. A benchmark that only asked "find the bug" would reward pattern recognition; requiring a working crash input and a non-breaking patch forces the agent to close the loop from suspicion to demonstrated defect to fix.

RDI hosts the benchmark inside a broader observatory that tracks AI cyber capability across attack and defence stages, and the framing is worth noticing: CyberGym is one instrument on a bench, alongside ExploitGym and the end-to-end variant. Reading it as a standalone leaderboard loses that context, which is one reason the numbers travel badly.

Four stages, three of which score

The scoring rule is the most under-reported part of the benchmark, because it is where a plausible-sounding score can diverge from what actually happened.

Stage 1 is the reproduction: does the agent's proof-of-concept input crash the unpatched build. Stage 2 is the patch: does the submitted change stop that specific crash. Stage 3 is regression: do the developer-written functionality tests still pass. Those three produce the score.

Stage 4 asks whether the patch fixes the actual ground-truth vulnerability, and it is recorded as a diagnostic rather than counted. That asymmetry deserves a moment, because it is the difference between "the agent can make a crash go away" and "the agent can fix the bug". Both are useful claims; they are not the same claim. An agent that suppresses a crash by catching the signal, widening a bounds check past the overflow, or disabling the code path entirely will score on stages 1 through 3 and fail stage 4. If a write-up reports a single number without saying whether stage 4 was consulted, you cannot tell which of the two things was measured — and the diagnostic exists precisely so that you can ask.

How the 131 tasks were chosen — and what that does to the score

The source dataset is larger than the scored set, and the filtering is the single biggest lever on comparability. RDI's dataset contains 920 vulnerabilities across 139 open-source projects, drawn from Google's OSS-Fuzz. The end-to-end variant used by third-party trackers narrows that to 131 tasks, with one task per project, selected by difficulty, excluding tasks with oracle leaks or sandbox compatibility constraints.

Three consequences follow directly. First, one task per project means the score is dominated by breadth rather than depth: an agent that is average across many projects will outperform one that is excellent on three, which is a defensible design choice and a different measurement from "how many vulnerabilities can this agent fix". Second, excluding oracle leaks removes tasks where the answer is discoverable from the environment rather than reasoned out, so the remaining set is harder in a specific way — it removes shortcuts rather than removing easy tasks. Third, a 90-minute budget per task inside a sandbox built from the project's OSS-Fuzz image is part of the definition of success, not a detail, because a large share of real vulnerability work is bounded by exactly that kind of constraint and agents that depend on unbounded exploration will fail here while succeeding elsewhere.

Any comparison that ignores the task-selection rule is comparing two different tests that happen to share a name.

Two implementations, and why they do not have to agree

Here is where most of the confusion lives. CyberGym-E2E exists as an RDI benchmark and as an implementation run by a third-party evaluator, and the two are not guaranteed to produce the same number for the same model. The evaluator's own documentation says so plainly: its implementation uses a filtered set of 131 tasks, one per project, chosen by difficulty, and it runs every task in an isolated sandbox built from the project's OSS-Fuzz image using its own agent harness with a 90-minute budget per task.

That sentence contains the two variables that decide the score. The harness is the loop the agent runs inside — how it observes tool output, how many steps it can take, how it recovers from a failed attempt. A stronger harness with a weaker model can beat the reverse, and the harness is not part of the model's weights. The budget interacts with both. So a CyberGym figure is properly written as "this model, in this harness, in this implementation", and a figure quoted without the implementation is a number without a denominator.

There is a second-order effect worth stating: because the harness is doing real work, a CyberGym score improvement between two model versions may partly be a harness improvement, and a vendor that reports its own harness results is reporting system capability rather than model capability. Both are legitimate; conflating them is not.

Reading the tracker table honestly

The table most people encounter is a secondary compilation, and it labels its own provenance — which is the useful part, if you read the label. Of the 29 rows, 7 come from the benchmark owner or an independent run, 21 are provider self-reports, and one is a secondary report. The compilation states outright that it shows the table for reference and does not rank on it, and advises reading it with care because most rows are self-reports. Its own saturation note is the most useful line on the page: the top three are spread across about 70 points while the top ten sit within roughly 10, so the table still separates models — but not at the top of the range, where the ordering is thin.

A provider self-report is displayable evidence. It is not an independent measurement, and the difference matters in this benchmark more than in most, because the harness variable gives a vendor considerable room to tune the number it reports. When a self-report and an independent run disagree, the independent run wins. When only a self-report exists, the honest reading is "unverified, plausible, no comparable baseline".

The same caution applies in reverse to the trap this benchmark sets for readers: a model with no CyberGym row has not scored zero. It has not been measured in that harness, which is a different statement. Treating absence as failure is how a table stops being a measurement and becomes a leaderboard.

What a high score does and does not mean for a defender

The question a security team actually has is not "which model tops the table" but "what does an agent that scores well on this do to my risk". The benchmark answers a narrower question than the one being asked, and the gap is worth naming.

It does mean the agent can, within 90 minutes and inside a known build, locate a memory-safety bug in real open-source code, produce a functioning crash input, and write a change that stops the crash without breaking tests. That is a genuine capability, it is directly relevant to triage and patch work, and it is the kind of task where a competent agent meaningfully compresses an analyst's queue.

It does not mean the agent understands your codebase, your threat model, or your deployment. The tasks are drawn from public OSS-Fuzz targets with a known ground-truth vulnerability and a test suite that tells the agent whether it is done. Production code has no such oracle. It also does not mean the agent will find the bug rather than a bug: stage 4 exists because those diverge, and stage 4 does not count.

And it does not transfer to the offensive side, which is where the same capability reads differently. An agent that can locate and reproduce a memory-safety flaw can do so in code it was not given permission to test, which is why capability of this order is exactly what motivates access control at the model level rather than a disclaimer in a system prompt.

What to ask before trusting a CyberGym number

Five questions, answerable from a well-written report and usually unanswerable from a launch post.

• Which implementation? RDI's benchmark, or a third-party evaluator's filtered variant. If the report does not say, the number is not comparable to anything.

• Whose run? Independent evaluation or provider self-report. The compilation table labels this per row; most marketing does not.

• Which harness, and what budget? If the harness is the vendor's own and is not described, treat the result as a system-level claim about a specific product configuration.

• Was stage 4 consulted? A reported rate with the diagnostic alongside it tells you whether patches fixed the bug or merely silenced the crash.

• Which task set? 131 one-per-project tasks chosen by difficulty is the common set; a different filter is a different test.

Cybergym is a well-constructed benchmark for a hard, real task, and its construction is more interesting than its leaderboard. The stages tell you what an agent had to do; the filtering tells you what the score is a score of; the harness tells you who deserves credit. A number that arrives with those three attached is genuinely useful. A number that arrives alone is a press release with a percentage sign.