Code Review Agent Benchmark: How to Evaluate a Reviewer, and Run c-CRAB on Your Own Code

How do you tell whether a code review agent is any good? For most of this field's short history the answer was "measure how close its comments are to a human reviewer's" — which sounds reasonable until you actually try it, because two reviewers can raise the same issue in completely different words. The Code Review Agent Benchmark — the paper is arXiv:2603.23448, its dataset is c-CRAB — is the first serious attempt to score a review by what acting on it produces rather than by its phrasing. It converted 234 human review comments into executable tests, ran four widely-used reviewers against them — PR-Agent, Devin, Claude Code, and Codex — and found that all four taken together pass 41.5% of those tests, "only around 40%" in the paper's own words. This page is a playbook: how to read that result without mangling it, how to run c-CRAB yourself, and what to do when your codebase is not in the benchmark at all.

The headline number is the least useful thing on this page. The useful things are the method and the failure modes: why every earlier scoring scheme was measuring the wrong thing, what it costs to score a review with executable tests instead, and why "review agents only catch 40% of bugs" is a three-fold misreading of the actual result. Everything here is a community reading of the published benchmark and of practitioners' experience running it — not vendor guidance from the toolmakers involved.

Why the obvious metrics do not work

Before c-CRAB, evaluations of code review agents fell into a small number of families, and the paper's own comparison table (Table 1) lays the lineage out. The oldest is text overlap — BLEU, ROUGE, chrF and friends, used by benchmarks like CodeReviewer and ContextCRBench. The idea is that an agent's comment is good when its n-grams match a human's. The idea collapses on the one kind of case that is everywhere in code review: the same defect described in different words.

The paper's case study is the cleanest example. On a pull request in python-telegram-bot (PR #3514), the human reviewer and Codex both flagged the same nested-indexing robustness bug. Codex's review was behaviourally correct — a coding agent that acted on it produced a fix that passed the executable test. Yet the text metrics scored it BLEU-4 0.00, ROUGE-L 7.02, chrF 20.74, and embedding similarity 54.59. Zero n-gram overlap, and the review was right. Same concern, different words: the string metrics could not see it. Embedding similarity is a partial step up — 54.59 against a confirmed pass is still nowhere near a usable threshold — and it inherits the same problem in softer form.

LLM-as-judge, where a model compares the agent's review to the human's and votes, fixes the vocabulary problem but imports three new ones, which the paper names directly: bias, instability, and sensitivity to prompt design. Run the same comparison twice and a judge can hand you different verdicts; rephrase the judging prompt and rankings move. When you are choosing between two reviewers that sit three points apart on the same benchmark, a judge with that variance cannot back a decision — and a score you cannot reproduce is not a score.

What an executable oracle buys — and what it costs

The idea c-CRAB is built on is simple and radical at once: instead of asking "does the review sound like the human's?", ask "if you act on the review, does the code get fixed?". Each retained human review comment is converted into an executable test that captures the underlying issue. A review comment counts as correct if acting on it produces a behaviourally correct fix that makes the test pass — and every instance ships with a runnable Docker environment, so "makes the test pass" is a fact rather than a judgement.

The paper defines two kinds of test. Behavioral tests "import and execute the tested code at runtime," invoking functions "with specific inputs" and checking "outputs or verify[ing] exceptions." Structural tests "inspect source code text, matches patterns, and check API surfaces to determine whether desired code changes have been made." The final split is 42 behavioural (17.9%) and 192 structural (82.1%) — and that skew deserves an honest sentence: most of this oracle is pattern-matching on source text, not executing the code. The gold standard is the behavioural test; the majority of the dataset is the pragmatic version of it.

Building the oracle is a four-stage funnel, and every stage throws things away:

• Initial dataset — 671 PRs, 1,313 review comments.

• Review filtering — 410 PRs, 595 comments. An LLM classifier, calibrated against a gold set of 100 manually annotated comments, keeps only objectively verifiable issues and drops conversational or subjective feedback.

• Executable environment construction — 410 PRs, 595 comments. One Docker image per PR, with dependency resolution falling back to a coding agent where automation fails.

• Converting NL comments to tests — 339 PRs, 481 comments. Generated with GPT-5.2 under an execution-guided refinement loop (up to three attempts); a test is kept only if it fails on the before version and passes on the after version.

• Validation with a coding agent — 184 PRs, 234 comments (final). Claude Code on a Sonnet-4.6 backend tries to fix the code given only the human review comment; instances where it cannot make the test pass are discarded.

About 27% of the starting pull requests survive. State that plainly, because it is the honest price of a test-based oracle: if a comment is not actionable enough to become a failing test, or the environment cannot be built, or a competent coding agent cannot fix the code from the comment alone, the instance is dropped. It is also why the benchmark is small. 184 PR instances and 234 validated comments is a dataset you can read, not a corpus you can drown in — and for an oracle that has to run real Docker environments, smallness is a feature.

For scale: an average instance touches 418.1 modified lines, tests average 31.8 lines, and there are 1.27 tests per instance. Two annotators agreed 84% of the time — over 50 sampled instances — on whether a generated test faithfully captured the human reviewer's concern.

One bibliographic wart you will hit if you go read the paper yourself: the dataset table (Table 4) lists 67 repositories, while the Threats to Validity section says "184 pull request instances with 234 verifiable oracles across 56 repositories." The paper gives both figures in different places and does not reconcile them. Do not pick a favourite and do not average them — cite each where it appears. Discrepancies like this are exactly the detail readers use to decide whether a benchmark is worth their time.

For due diligence on independence: the paper discloses that one author is affiliated with SonarSource, and states the findings should not be interpreted as "an evaluation of the quality of products at SonarSource." That is their disclaimer, quoted rather than paraphrased.

How to read a score on c-CRAB without misquoting it

The headline metric is the pass rate: per instance, the share of that PR's tests that pass, averaged across the 184 instances. Here is the full results table from the paper, one line per reviewer. The human row is a scale marker rather than a competitor — the humans wrote the oracle, so they score 100% by construction:

• Claude Code — 1,336 comments, 7.3 per PR, overall 32.1% (behavioural 38.1%, structural 30.7%).

• Devin — 1,344 comments, 7.3 per PR, overall 24.8% (behavioural 31.0%, structural 23.4%).

• PR-Agent — 524 comments, 2.8 per PR, overall 23.1% (behavioural 38.1%, structural 19.8%).

• Codex — 324 comments, 1.8 per PR, overall 20.1% (behavioural 38.1%, structural 16.1%).

• Human — 234 comments, 1.3 per PR, 100% by construction.

Three corrections, because the abstract's "only around 40%" is the most misquoted number in this corner of the AI-coders conversation right now. First, the 41.5% figure — 97 of the 234 tests passed by at least one tool — is a union across all four reviewers: a test counts once if any agent passed it. No single agent scored 41.5%; the best single score is Claude Code's 32.1%. Second, the human row is the oracle, not a contestant; repeating it as "humans beat the bots" is a category error. Third, and most important: c-CRAB gives no credit for a valid problem the human reviewer never raised. The oracle is human review intent. An agent that finds a real bug nobody mentioned scores zero for it. So "AI review agents only catch 40% of bugs" is wrong three times over — it is a union, it is not a bug-catch rate, and it measures agreement with human reviewers, not total correctness.

Comment volume is the trap

The most interesting number in the results is not the winner. Claude Code and Devin each posted more than 1,300 comments — about 7.3 per PR — to reach 32.1% and 24.8%. Codex posted 324 comments, about 1.8 per PR, and reached 20.1%. The human baseline is 1.3 comments per PR. Volume is not coverage: roughly five times the comments buys well under double the pass rate. If you are choosing a reviewer, the real cost of all those extra comments is human review fatigue — every comment an agent posts is a judgement call a person has to triage.

The usefulness finding cuts the other way, and it is the piece that keeps this from being a cheap "the bots are noisy" story. The authors hand-inspected 92 comments across 6 PRs and judged 84% (77/92) useful — PR-Agent 94%, Codex 88%, Devin 85%, Claude Code 78%. So most comments that fail the test are not noise; they are about something the human reviewer did not raise. The sample is small — 92 comments, 6 PRs — and worth saying in the same breath as the percentages.

What the two sides actually talk about explains the shape of the results. Human reviewers skewed toward maintainability, design, and documentation; the tools skewed toward robustness, testing, and error handling. The paper reads this as an argument for human-agent collaboration rather than replacement — and it is also the best available explanation for why the scores look low. A reviewer that is sharp on edge cases but quiet on design will systematically miss the categories humans flag, and the oracle is built entirely from human flags.

Practitioners who have worked through this come to the same place. A detailed write-up by Daniel Vaughan that calls the work CR-bench reaches the same conclusion and turns it into a workflow: let the agent do the robustness and correctness sweep, keep humans on design, conventions, and architecture — the categories where agents score worst — and steer the agent with review instructions that name the weak categories. His most useful caveat for anyone reading the leaderboard: "usefulness is not the same as pass rate," because the test suite requires matching the human's intended fix, and a valid alternative fix fails the test. The path from 20% to a meaningfully higher score, in his read, is not a model upgrade — it is configuration work.

Running c-CRAB yourself

Everything above is reading other people's results. The replication package makes the benchmark runnable — it lives at c-CRAB-Benchmark/dataset on GitHub — and the README is honest about what it takes.

Requirements: code>uv sync/code>; Docker; and either code>OPENAI_API_KEY/code> or code>ANTHROPIC_API_KEY/code> (Claude Code additionally reads credentials from code>~/.claude/.credentials.json/code>, mounted into containers by default). The layout is five directories: code>pipeline//code> (pipeline logic and prompts), code>execution//code> (Docker image builders and runtime helpers), code>results_preprocessed//code> (the released benchmark subset), code>results_pipeline_funnel//code> (the stage0–stage4 JSONL files and funnel summary), and code>raw_results_compressed//code> (raw experiment outputs). The five steps, in order:

1. Build the Docker environments — code>uv run python -m execution.build_swe_care --split test --instance results_preprocessed/instance-ids.txt --max-workers 4/code>. Prebuilt images are also published under the c-CRAB-Benchmark GitHub packages org if you would rather skip the build.

2. Generate the tests — code>./run_testgen_full.sh --instances-file results_preprocessed/instance-ids.txt --workers 4 --output-dir results_testgen/code>.

3. Collect baseline reviews — code>uv run python run_batch_baselines.py --split test --instances-file results_preprocessed/instance-ids.txt --tools pr-agent devin claude-code codex --output-dir baselines_output --workers 4/code>. Configure the corresponding external tool credentials before this step.

4. Run agent resolution — code>uv run python run_batch_agent_resolution.py --stage3-file results_pipeline_funnel/stage3_testgen_verified.jsonl --testgen-dir results_testgen --output-dir results_agent_resolution --workers 4/code>.

5. Evaluate — repeat once per tool: code>uv run python run_batch_tool_eval.py --tool pr-agent --stage3-file results_pipeline_funnel/stage3_testgen_verified.jsonl --testgen-dir results_testgen --tool-results-dir baselines_output --output-dir results_eval_pr-agent --workers 4/code>.

Two things the README does not advertise. Adding a fifth reviewer means editing code>run_batch_baselines.py/code> — that is where the per-tool baseline review prompts live, and there is no plugin interface; the README documents no cleaner extension point. And the repository carries no explicit licence file, so do not assume the code is MIT or Apache — the paper is CC BY 4.0, and the code's own terms are unstated.

Cost is the other unadvertised item. The paper publishes no token or dollar figures for running the pipeline, so treat any cost number you see quoted online as unverified. What the structure implies is clear enough: one Docker image per PR across 184 instances, plus a coding-agent resolution pass and an evaluation pass per tool. That is not a laptop-scale afternoon — budget for real compute.

When you cannot afford executable oracles

The honest position of most teams is: the benchmark is right that an LLM judge cannot score reviews, but building a test-based oracle for your own PRs is a big lift. The distinction worth drawing is between an LLM judge as a scorer and an LLM judge as a filter. c-CRAB's rejection of the judge as an oracle does not make a judge useless inside a reviewer — a judge that clusters duplicate findings and drops weak ones can still raise precision. The failure mode to design against is independence.

A judge that runs on the reviewer's own model agrees with itself: it reads the review, finds it plausible, and reports success while changing nothing. A different-vendor judge reduces that self-agreement — it does not turn a judge into a test, but it stops the rubber stamp. We can show you a concrete, checkable instance of exactly this guardrail because our own harness is open: Orca-Code-Review on GitHub is MIT-licensed, and its routing recipe states the rule in the repo's own words — the judge "MUST NOT NAME THE DEFAULT'S MODEL," because "on the reviewer's own model it agrees with itself, so the pass goes inert while still reporting success." The Action never names a model; the recipe decides. As provisioned, the reviewer default is deepseek/deepseek-v4-flash-0731, and a rule matching the code>x-cr-lens: judge/code> header sends the judge pass to z-ai/glm-5.3 — a different vendor. That is a design parallel to c-CRAB's argument, not a result: we are not in the benchmark and there is no c-CRAB score for our reviewer. But it is the practical mitigation available to anyone who cannot build executable oracles, and it is cheap when judge and reviewer can live on different providers behind one key — which is what a router is for. On OrcaRouter the reviewer and its judge are two lines in a routing DSL, and you pay provider list price with zero markup.

When your code is not in the benchmark

184 PRs across 56-or-67 public repositories is not your codebase, and it was never going to be. The transferable part is the method, and you can run it on your own history at a much smaller scale. Take merged PRs that had human review comments. For a sample of those comments, write a test that fails before the review was acted on and passes after — the fail-then-pass property is the whole game. Run your candidate reviewer on the pre-review diff. Then check whether acting on its comments makes the test pass. What you get is a number computed on the code you actually ship, which is worth more than a leaderboard position. What it costs is exactly the wall the paper hit: you need reproducible per-PR environments, because a test that only passes on your laptop is not an oracle.

You do not need 234 tests. A dozen well-chosen ones on PRs your team actually argued about will tell you more about your reviewer than a benchmark score will. And a parallel practitioner analysis of this benchmark family is blunt about the gate: the LLM classifier's precision on whether a comment is a valid, verifiable issue lands between 66% and 85%, so treat machine filtering as a shortlist and keep a human adjudication step before anything becomes a test. The same write-up notes that LangChain's ReviewBench, built independently on the same comment-to-test idea, recovers roughly 30% of its baseline issues at best — the same ballpark as c-CRAB's 20–32%, and a reminder that single-digit leaderboard deltas between tools are often smaller than the noise in your own setup.

If you are deciding which review tool to buy at all, that is a different question — our code-review-agent buyer's guide covers bot-versus-agent, per-seat versus per-token pricing, and when self-hosting wins — and once you have one, the running cost of a review harness on every push is covered in our automated code review explainer. This page is only about measurement, and the companion piece to this one walks through the benchmark's anatomy: the construction funnel, the dataset statistics, and the full results table.

FAQ

Is 41.5% the best agent's score? No. 41.5% is the union across all four tools — a test counts once if any of them passed it. The best single score is Claude Code's 32.1%.

Does c-CRAB measure how many bugs a reviewer catches? No. It measures how well a review matches what a human reviewer raised, converted into executable tests. A real defect the human never mentioned scores zero, however valid it is.

Did the human reviewers "beat" the bots? The 100% human row is the oracle itself — the humans wrote the tests — so it is a scale marker, not a competitor.

Is c-CRAB the same thing as CR-bench? Yes. The dataset is c-CRAB; some third-party coverage calls it CR-bench, but there is only one benchmark here.

What does running it cost? The paper publishes no cost figures. One Docker image per PR across 184 instances, plus an agent-resolution pass, implies real compute — not a laptop-scale afternoon.

Bottom line

c-CRAB's contribution is not the leaderboard — it is the demonstration that a review can be scored by executing its advice, and that the text-similarity and LLM-judge schemes that came before were scoring the wrong thing. If you take one thing away, make it the three-part correction: 41.5% is a union, the human row is the oracle, and the benchmark gives no credit for defects humans never raised. And if you want a number you can act on, the method transfers — fail-then-pass tests on your own merged PRs, a human adjudication step, and, if you cannot build executable oracles, at least a judge whose model is independent of the reviewer's.

If you would rather measure a reviewer than argue about one, start from a harness you can read. OrcaCode Review runs a review pass plus an independent verification judge, per token rather than per seat, and every prompt in it is public — so you can point it at a benchmark like this one and get your own number instead of ours.