c-CRAB, the Code Review Agent Benchmark: What It Measures, What It Found, and What 41.5% Really Means

A code review agent and a human reviewer looked at the same pull request and raised the same concern. Acting on the agent’s comment fixes the bug; the test passes. And every text-similarity metric the authors computed scored the agent’s review as essentially unrelated to the human’s: BLEU-4 0.00, ROUGE-L 7.02, chrF 20.74, embedding similarity 54.59. Same concern, different words, and the standard way of scoring a review could not see that they agreed. That one example is the argument behind c-CRAB (pronounced “see-crab”), the Code Review Agent Benchmark published as arXiv:2603.23448.

c-CRAB evaluates code-review agents, not code-writing agents. Given a pull request — which may come from a human or from a coding agent — a review agent produces a review, and c-CRAB scores that review by whether acting on it produces a behaviourally correct fix. The benchmark was built by software-engineering researchers Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, Haifeng Ruan, Ridwan Shariffdeen and Abhik Roychoudhury, and it evaluates four tools: PR-Agent, Devin, Claude Code and Codex. One of the authors is affiliated with SonarSource, and the paper is explicit about what that does and does not mean, in its own words: “The views and conclusions expressed in this paper are those of the authors alone and do not represent the official policies or endorsements of SonarSource. Furthermore, the findings presented herein are independent and should not be interpreted as an evaluation of the quality of products at SonarSource.”

Two notes before the detail. Some third-party write-ups call the same work “CR-bench”; it is the same benchmark, and this page uses c-CRAB throughout. Every figure below is the paper’s own reported result, read today from the paper and the replication package — not independently re-run — and the interpretation is ours plus the practitioner discussion the paper has already attracted. None of it is guidance from the vendors whose tools were evaluated. And if you are still deciding whether to run a code-review agent at all, our buyer’s guide on code-review agents is the better starting point; this page is about how those agents are measured.

Why c-CRAB scores reviews with tests, not an LLM judge

The usual way to score a code-review agent is to compare its review to a human’s, with an LLM-as-judge or a text-similarity metric. c-CRAB’s authors reject both. LLM-as-judge, they argue, suffers from bias, instability and prompt sensitivity, which makes reproducible, consistent scoring hard. And the case study above shows what string metrics actually measure: wording, not effectiveness. On that python-telegram-bot pull request, the Codex review said the same thing the human did, and BLEU-4 and ROUGE-L could not recognise it.

So c-CRAB does the opposite. Each human review comment is converted into an executable test that captures the underlying issue. A review comment counts as correct if acting on it produces a behaviourally correct fix — one that makes the test pass. Every instance ships with an executable Docker environment, so the pass/fail decision is made by running code, not by asking another model how similar two texts are. That is why it matters: a review’s job is to change what a developer does, and a test is the only scoring signal that measures that change directly.

The paper defines two kinds of tests, in its own words: “Behavioral tests import and execute the tested code at runtime. They invoke the tested functions with specific inputs, and check outputs or verify exceptions. On the other hand, structural tests inspect source code text, matches patterns, and check API surfaces to determine whether desired code changes have been made.” The final split is 42 behavioural (17.9%) and 192 structural (82.1%). It is worth a sentence of honesty: most of the oracle is pattern-matching on source text, not executing the code. That skew is a real limitation to keep in mind.

How the benchmark was built, and what the funnel cost

c-CRAB is built on top of the existing inclusionAI/SWE-CARE dataset, which supplies pull-request instances with commit metadata; c-CRAB’s own contribution is the oracle, not the PR corpus. The curation pipeline runs four filters, and each one costs instances. The paper reports the funnel as follows:

• Initial dataset — 671 PRs, 1,313 comments.

• Review filtering — 410 PRs, 595 comments. An LLM classifier, calibrated against a gold set of 100 manually annotated comments, keeps only objectively verifiable issues and drops conversational or subjective feedback.

• Executable environment construction — 410 PRs, 595 comments. One Docker image per PR, with dependency resolution falling back to a coding agent where automation fails.

• Converting natural-language comments to tests — 339 PRs, 481 comments. Tests are generated with GPT-5.2 under an execution-guided refinement loop of up to three attempts; a test is kept only if it fails on the original code and passes after the fix.

• Validation with a coding agent — 184 PRs, 234 comments. Claude Code on a Sonnet-4.6 backend tries to fix the code given only the human review comment; instances where it cannot make the test pass are discarded. This is the final set.

About 27% of the starting pull requests survive. That is the honest price of a test-based oracle, and it is also the reason the benchmark is small rather than sprawling. The surviving set: 184 PR instances, 234 validated review comments, 1.27 tests per instance, 418.1 modified lines per PR on average, 31.8 lines per test. Two annotators independently judged whether a generated test faithfully captured the human reviewer’s concern, on 50 sampled instances, and agreed 84% of the time.

One discrepancy you will notice if you read closely: the dataset table lists 67 repositories, while the threats-to-validity section says “184 pull request instances with 234 verifiable oracles across 56 repositories.” The paper gives both figures, in different places, and we are not going to average them or silently pick the convenient one. Readers use exactly this kind of detail to judge whether a benchmark is worth their time, so both are reproduced here as published.

The results, and how to read them

Pass rate is the aggregate test pass rate: per instance it is the share of that PR’s tests that pass, and the headline is the average across instances. The paper reports, per tool:

• Claude Code — 1,336 comments, 7.3 per PR — behavioural 38.1%, structural 30.7%, overall 32.1%

• Devin — 1,344 comments, 7.3 per PR — behavioural 31.0%, structural 23.4%, overall 24.8%

• PR-Agent — 524 comments, 2.8 per PR — behavioural 38.1%, structural 19.8%, overall 23.1%

• Codex — 324 comments, 1.8 per PR — behavioural 38.1%, structural 16.1%, overall 20.1%

• Human — 234 comments, 1.3 per PR — 100% by construction. The humans wrote the oracle, so this row is a scale marker, not a competitor.

Read those rows carefully before quoting any of them. The “only around 40%” from the abstract is a union: 41.5% of the 234 tests were passed by at least one of the four tools. It is not any single agent’s score — the best single score is Claude Code’s 32.1% — and it does not mean the four tools together caught 40% of real defects. The section below explains why.

The most interesting number is not the winner. Claude Code and Devin each posted more than 1,300 comments — about 7.3 per PR — to reach 32.1% and 24.8%. Codex posted 324, about 1.8 per PR, to reach 20.1%. The human baseline is 1.3 comments per PR. Do the arithmetic: roughly four times the comment volume buys well under double the pass rate. Volume is not coverage. A chatty reviewer is not the same thing as a useful one, and c-CRAB is the first benchmark set up to show that.

Usefulness cuts the other way

The low pass rates read as condemnation until you look at what else the authors measured. They hand-inspected 92 comments across 6 PRs and judged 84% of them useful (77 of 92) — PR-Agent 94%, Codex 88%, Devin 85%, Claude Code 78%. So most comments that fail a c-CRAB test are not noise; they are about something the human reviewer did not raise. The sample is small — 92 comments, 6 PRs — and the paper says so, and so should we.

The same pattern shows up in what the reviewers talk about. Human reviewers skewed toward maintainability, design and documentation; the tools skewed toward robustness, testing and error handling. The paper reads this as an argument for human-agent collaboration rather than replacement. It is also the best available explanation for why the scores look low: agents and humans are often not looking at the same things, and the oracle only rewards the human’s list.

What c-CRAB cannot see

The benchmark is explicit about its blind spot, and so are we. c-CRAB gives no credit for a valid problem the human reviewer never raised. The oracle is human review intent: an agent that finds a real bug nobody mentioned scores zero for it. The paper puts it directly — automated review tools may generate other valuable comments that human reviewers did not identify, but “like other existing benchmarks, c-CRAB does not directly evaluate these additional comments.”

That single sentence is the correction to most coverage of this result. Anyone quoting “review agents only solve 40%” as if it measured how many real defects the agents catch is misreading the number. It measures how many human-raised concerns the agents, jointly, managed to resolve — a narrower and much more honest claim.

Running it yourself

If you want to reproduce the numbers or add your own reviewer, the replication package is public at c-CRAB-Benchmark/dataset. The README is the actual documentation, and it is honest about the shape of the thing. Setup is code>uv sync/code>; you need Docker and an code>OPENAI_API_KEY/code> or code>ANTHROPIC_API_KEY/code>, and Claude Code additionally reads credentials from code>~/.claude/.credentials.json/code>. The org also publishes prebuilt Docker images for the environments.

The layout: code>pipeline//code> holds the pipeline logic and prompts, code>execution//code> the Docker image builders and runtime helpers, code>results_preprocessed//code> the released benchmark subset (410 preprocessed instances), code>results_pipeline_funnel//code> the stage0–stage4 JSONL files and the funnel summary, and code>raw_results_compressed//code> the raw experiment outputs.

Reproducing the full run is five steps: build the Docker environments (code>execution.build_swe_care/code>), generate the tests (code>run_testgen_full.sh/code>), collect baseline reviews (code>run_batch_baselines.py --tools pr-agent devin claude-code codex/code>), run agent resolution (code>run_batch_agent_resolution.py/code>), then evaluate (code>run_batch_tool_eval.py --tool <name>/code>). If you want to add a fifth reviewer, be aware the extension point is not a plugin interface: the baseline-review prompts for every tool live in code>run_batch_baselines.py/code>, and the README does not document a cleaner way — you edit that script.

Two more facts before you clone it. The paper is licensed CC BY 4.0; the repository page does not state a licence for the code, so do not assume one. And the paper publishes no cost or token-usage figures for running the benchmark — that is not published, so we are not going to invent it. What the pipeline does imply: one Docker image per PR across 184 instances, plus an agent-resolution pass, is not a laptop-scale afternoon.

What this means for anyone who ships a review pipeline

c-CRAB’s core argument is that an LLM judge is an unreliable oracle. If you cannot build executable oracles — and most teams cannot — the best available mitigation is to never let the judge run on the model that produced the review. A judge that shares the reviewer’s model agrees with itself, and the verification pass turns into a rubber stamp that still returns a number.

That is exactly the failure the routing recipe behind the reviewer we ship guards against — and it is a design parallel with c-CRAB’s critique, not a benchmark result. The harness runs an LLM judge anyway, as a second pass that clusters findings, scores each cluster 0–1 for whether it is a concrete defect in this change, and drops everything under a threshold. The recipe that governs it, code>recipes/orcacode-review.dsl.yaml/code>, is a public file. The Action never names a model: it calls a router alias, and the recipe decides. As provisioned, the recipe is four lines — the reviewer default is code>deepseek/deepseek-v4-flash-0731/code>, and a rule matching the header code>x-cr-lens: judge/code> sends the judge pass to code>z-ai/glm-5.3/code>, a different vendor. The recipe’s own words require that the judge “MUST NOT NAME THE DEFAULT’S MODEL”, because on the reviewer’s own model it “agrees with itself, so the pass goes inert while still reporting success”.

A different-vendor judge reduces self-agreement; it does not turn an LLM judge into a test. c-CRAB did not test our reviewer, and we are not going to imply otherwise. OrcaCode Review runs a review pass plus an independent verification judge, per token rather than per seat, and every prompt in it is public — so you can point it at a benchmark like this one and get your own number instead of ours.

Bottom line

c-CRAB is the first code-review benchmark whose scoring you can mostly trust to mean what it says: a review passes only when acting on it fixes the code. The headline numbers are genuinely low — best single tool 32.1%, union 41.5% — but they measure overlap with human-raised concerns, not the quality of the reviews, and the usefulness data shows most comments are real signal. The durable takeaways are the ones the paper itself argues for: volume is not coverage, agents and humans look at different things, and the right deployment is human-agent collaboration. And the benchmark is open, so the honest next step is to run your own reviewer on it and get your own number.