Automated Code Review in 2026: Make It Run on Every PR Without Buying a Seat

Automated code review is a CI job that sends your diff to a language model, posts findings on the affected lines, and fails a status check when it finds something serious. The way to get that running on every pull request without a per-seat subscription is to self-host an open-source harness: copy a roughly fifteen-line workflow into your repository, add one API key, and pay only for the tokens each review consumes. There is no seat count to buy, because there is no seat. The reference implementation we maintain is the Orca-Code-Review repository — public, MIT-licensed, and since its creation on June 25, 2026 the code behind the OrcaCode Review GitHub Action. Its shipped routing recipe defaults the review pass to DeepSeek V4 Flash and the independent verification judge to GLM-5.3, and both are yours to change. This article walks through what actually runs on every push, what you configure, what it costs in tokens, and the failure modes you will meet in week two.

The short version. Every push gets one review. Findings post inline on the changed lines. P0 and P1 findings fail the check and block the merge; a clean run passes. You re-review on demand by commenting /orcacode-review. The workflow lives in your repo; the review logic lives in the published action; the model choice lives in a routing recipe you can edit in your own workspace. The reviewer reads the diff and repository files and never executes your PR’s code. And the honest caveat up front: it catches real bugs and still misses the ones that need a human who knows why the code is the way it is.

• One workflow + one secret + a token bill. No per-seat license at any point.

• The harness is open source. Copy it, fork it, audit it, pin it to a commit SHA.

• The model is a setting, not a vendor. Change the reviewer by editing a routing recipe, not by rewriting YAML or bumping the action.

• Oversized diffs cost nothing. The size guard runs before the model does.

• It reads, never runs, your code. That is the safety property that makes pull_request_target safe to use at all.

How automated code review actually works

Every automated review system is the same three ingredients wearing different clothes: an event, a runner, and a reviewer.

The event is the trigger. The shipped workflow fires on pull request events — opened, synchronize (a new push), ready_for_review (a draft becomes ready) — and on a PR comment. Because it runs on pull_request_target, the workflow definition is read from the base branch, which is why the workflow must exist on the base branch before it can run for a PR. One review per push; the concurrency block cancels the previous run, so a rapid sequence of pushes does not queue five reviews of stale code.

The runner is GitHub Actions on ubuntu-latest. The job needs three permissions: read access to contents, write access to pull requests (to post inline comments), and write access to issues (to post the summary and clean up stale comments).

The reviewer is a language model. The action fetches the PR head, assembles the diff and the repository context the engine selects, and sends that to the review model. The result is a set of findings, each tagged with a severity and anchored to a file and line. The action posts them as inline PR comments and writes one summary comment into a marker region at the top of the PR description, replaced in place on every push.

The gate is a status check. GitHub does not know what “review” means; it only knows whether the review check passes. You make the gate real by marking that check as required in branch protection. That is the entire merge-blocking mechanism — no admin API calls, no labels, just a failing required check.

What does not happen: nothing executes the PR’s code. The engine only reads. That single invariant is what makes the privileged pull_request_target trigger safe to use with a paid API key.

The open-source harness is the differentiator

Everything above is true of many tools. What is not true of most is that the whole thing is inspectable and self-hostable, which is what the Orca-Code-Review repository buys you. It is a public MIT-licensed GitHub repository (JavaScript, created June 25, 2026) that packages the review as a reusable composite GitHub Action plus an installer, and it is the same code the hosted OrcaCode Review app runs.

Spend ten minutes in the tree and you can name every piece that touches your PR:

• action.yml — the composite action, about fifteen documented inputs. No model names are hard-coded anywhere in it.

• workflows/orca-code-review.yml — the example consumer workflow, the roughly fifteen lines you copy into .github/workflows/.

• recipes/ — the routing DSL. This is where the model is actually chosen.

• rules/ — the severity rubric (P0–P3), the mandatory output shape, and a conventions directive that feeds the project’s own conventions doc into the review as untrusted reference data.

• scripts/ — the precision filter (L1 plus an L2 judge), the diff guard, the merge gate, the run report, and the token meter. Each is a small, readable .mjs file with tests.

• skills/setup-orca-code-review — the skill the installer drops into your coding agent, covering install, reconfigure, troubleshoot, and uninstall.

• .claude-plugin/ — what lets Claude Code install the skill as a self-updating plugin.

Installing is a one-liner that teaches your AI what the product is and then stops:

npx @orcarouter/code-review

The CLI detects which coding agents you use — the catalog spans 36 platforms, from Claude Code, Cursor, Codex, OpenCode, and Windsurf to GitHub Copilot, Gem​ini CLI, Amazon Q Developer, Cline, RooCode, and others — installs the skill, and hands off. You then ask your agent in plain language: “set up OrcaCode Review in this repo,” “block P0 only,” “why didn’t the review run?” The skill carries the lifecycle: it writes the workflow, walks you through the API key, sets the gate, and asks only the questions that are genuinely yours.

Claude Code can install the skill as a plugin instead, which keeps it updated as the repo moves:

/plugin marketplace add Continuum-AI-Corp/orca-code-review

/plugin install orca-code-review

No agent at all? The same lifecycle is plain subcommands — init writes the workflow, reconfigure changes blocking rules and diff limits, doctor diagnoses reviews that don’t run or don’t post, uninstall removes it (dropping the required check first). The skill is the front door, not the only door. Or wire it by hand: copy the workflow, add one secret named ORCAROUTER_API_KEY, and mark the review check required.

The engine underneath is Ali​baba’s Open Code Review, pinned by exact version and Apache-2.0 licensed. OrcaCode decides how to review; OrcaRouter decides what model runs it. The self-host-versus-hosted ledger — what “free” actually costs when you self-host an open-source reviewer — is worked through in our article on open code review.

What runs, in order, on every push

It helps to know the order, because each step can fail or skip independently:

• The diff guard runs first, before the model ever does. If the merge-base diff exceeds 512 KB or touches more than 300 files, the review is skipped and a notice posts. The default is on-oversized-diff: fail, so a diff padded past the limits cannot walk through a required gate unreviewed. This is also the spend control: an oversized PR costs zero tokens.

• The engine reviews the diff. One pass, per-file concurrency defaulting to 24, with a wall-clock ceiling of 20 minutes per pass.

• The precision filter post-processes the raw findings. L1, a deterministic filter, verifies each finding’s claimed existing-code snippet against the reviewed commit and re-homes or drops mismatches. L2, an LLM judge, clusters findings by root cause and drops low-confidence clusters. Both layers are soft-fail: an error keeps the prior stage’s findings and never aborts the review.

• The gate applies. P0 and P1 findings fail the check; the PR summary counts every finding, including ones muted from the diff.

• The meter prints what it cost. The meter input records per-call token accounting — prompt, completion, cached tokens, and the model the router resolved — and prints a totals table in the job log.

• An optional run report sends severity counts and gate metadata to the OrcaRouter control plane for the analytics dashboard. It carries no code, no diff, and no finding text.

What you actually configure

There are three surfaces, and they have very different blast radius.

1. The workflow file. The consumer workflow is deliberately thin. The inputs worth touching live in the action: block-on (which severities fail the check — default P0,P1), fix-first (which severities stop an exhaustive review early), auto-review-authors (an allowlist for who gets auto-reviewed), max-diff-kb and max-diff-files and on-oversized-diff (the size guard), timeout-minutes, concurrency, meter, and report. Every one has a documented default, so a fresh workflow is five lines of YAML plus a secret.

2. The dashboard. With settings: true (the default), each run fetches per-repository settings from OrcaRouter → Apps → OrcaCode Review: the model, review mode, merge policy, report severities, quiet mode, exhaustive review, a custom rubric, and guardrails. Set settings: "false" and the workflow file is authoritative — no dashboard value can override it. If you never open the console, you lose none of the harness; you just configure in YAML.

3. The routing recipe — the one people miss. The action never names a model. Instead it injects raw facts as request headers — which tier the run was recorded as, whether the previous pass found a P0/P1, and a lens marker when the request is the L2 judge — and the workspace router’s DSL recipe maps those headers to a concrete model. The shipped recipe defaults the review to DeepSeek V4 Flash and the judge to GLM-5.3, routing the two to separate models on purpose. Changing the model that reviews your code is an edit to that recipe in your own workspace: no action version bump, no YAML rewrite, no redeploy.

The severity contract is two independent settings, not one. Merge policy decides what blocks the merge; report severities decides what gets posted on the diff. The shipped defaults are P0/P1 block, P2/P3 pass. A severity that blocks is always posted, whatever the report setting says — a failing check with nothing on the diff explaining it is worse than a noisy one. P0 means an exploitable security flaw, data loss, a crash on a normal path, or a broken build; P1 means a real but contained bug; P2 means a genuine defect that only fires under an abnormal precondition; P3 is style. When torn between two levels, the rubric says pick the lower one.

What it costs

Per token, not per seat. You choose the model on OrcaRouter, billing runs per token consumed, and the meter makes the per-run number visible instead of mysterious. The GitHub mechanics, Copilot’s metered code review since June 1, 2026, and how third-party reviewers slot into that workflow are covered in our GitHub code review guide. The full field-by-field cost comparison — per-seat products vs per-token ones, with a worked example — is in our AI code review tools comparison, and the question of what a single review pass costs in tokens when the reviewer actually explores the repository (the bot-vs-agent distinction) is in our code review agents piece. The point this article adds is the shape of the bill: it scales with the code you review, not the headcount that reviews it.

Two spend controls matter on day one. On a public repository, pull_request_target bypasses GitHub’s fork-approval gate, and the review key is wallet-metered — a stranger can open a PR and trigger paid reviews. Set a wallet budget with alerts on the key, and set auto-review-authors to something like OWNER,MEMBER,COLLABORATOR,CONTRIBUTOR so unknown contributors are not auto-reviewed. And the diff guard, as noted, means oversized PRs cost nothing at all.

What breaks

Automated review is CI. It breaks like CI, and the failure modes are mostly not the model’s fault:

• The workflow never runs. For pull_request_target the workflow is read from the base branch — a workflow added only in the PR branch will not run until it merges. Also check the app is enabled, auto_review is on, the PR is not a draft (drafts are skipped in ready_for_review mode), and Actions are enabled on the repository (forked repos ship with them off).

• /orcacode-review does nothing. The comment trigger requires the comment to start with one of four spellings — /orcacode-review, /orcacode review, @orcacode-review, @orcacode review — and the commenter to be an OWNER, MEMBER, or COLLABORATOR. A leading space breaks the match. An outside contributor’s command is silently ignored, on purpose: the command runs a privileged workflow holding the paid key.

• An auth error. The secret is misnamed or missing, the key is revoked or out of budget, or the workflow was switched to pull_request (which cannot read secrets from forks).

• The check is red with a “diff too large” notice. That is the size guard, working as configured. Split the PR, or raise the limits, or set on-oversized-diff: pass — and understand that with a required check, pass means a big enough PR walks straight through the gate unreviewed.

• The review runs but no comments appear. Three causes, all benign or configured: a clean run posts a summary rather than inline comments; quiet mode is muting P2 at posting time (the gate and report still counted it); or the precision filter dropped the findings — L1 drops findings whose snippet doesn’t match the commit, L2 drops low-confidence clusters. The job log’s severity counts tell you which.

The security posture is worth stating plainly because it is what makes the whole design safe. The engine only reads the diff and repository files; it never executes PR code. The reviewer has no merge authority — findings can block a merge or add a comment, but no code path lets model output approve or mutate the repository. An untagged finding fails safe, treated as blocking rather than advisory. And the run report carries no code or finding text. The two-layer setup that catches what a single-pass review misses is the subject of our AI code review security piece; the threat model above is documented in the repository’s SECURITY.md.

When automated review is the wrong tool

It is wrong more often than the tooling vendors admit. Sit this one out when:

• The problem is context, not volume. If reviews are slow because reviewers must understand why the code was written this way, an LLM reading the diff adds little. It has no memory of last month’s thread and no sense of the system’s history.

• The diff is mostly generated or vendored code. Auto-formatted output, scaffolded files, dependency snapshots. Reviewing it burns tokens and produces noise, and it is exactly where the conventions directive helps least — the code isn’t the project’s style by choice.

• The team already pair-reviews everything. Automated review is a volume lever. If every change is already reviewed by a human who was in the room, the machine adds a second opinion that is usually less informed than the first.

• Nobody reads the findings. A review nobody acts on is a workflow that fails green forever. This is the most common silent failure, and no precision filter fixes it.

• The review must run the code. If what you need is a test suite against the PR, an LLM review is the wrong tool. It reads; it does not execute. A security scan that needs to build and run the artifact belongs in a separate, carefully scoped job — remember, the review workflow must never be extended to run PR-controlled code.

• The repository is tiny or throwaway. Below a certain change rate, the review is more overhead than the bugs it catches.

False positives, and what precision filtering does and doesn’t fix

The accusation against every AI reviewer is that it cries wolf. The harness attacks this in two layers, and it helps to be precise about which layer fixes which failure.

The deterministic layer (L1) kills the ghost finding: an engine sometimes claims code that isn’t there — a snippet that drifted, a finding copied onto a sibling file. L1 verifies each finding’s existing-code snippet against the actual reviewed commit and re-homes or drops mismatches. That fixes the “this line doesn’t even exist” class of false positive, which is mechanical and verifiable.

The judge layer (L2) kills the duplicate and the unsupported claim: an LLM judge clusters findings by root cause and drops clusters whose confidence falls below the judge threshold (default 0.5). That fixes the “same bug reported three ways” and the speculative finding.

What neither layer fixes is worth saying out loud. A wrong but confident finding survives the judge — the judge is an LLM, and an LLM that sounds sure is not the same as a finding that is true. A judge that runs on the reviewer’s own model agrees with itself and the pass goes inert while still reporting success, which is why the shipped recipe routes the judge to a different model than the reviewer. And the severity rubric is deliberately conservative — “when torn between two levels, pick the lower one” — which means a real but conditional bug is more likely to land as a P2 advisory than a blocking P1. That is the right calibration for a tool that must not block everything, but it is a calibration: it trades missed blockers for fewer false alarms. The PR summary always counts every finding, so the muted P2s are still there to read. If the trade-off is wrong for your team, the rubric and the judge threshold are configuration, not a support ticket.

The bottom line

For a team that already lives in GitHub Actions, the open-source harness is the cheapest way to get automated code review on every PR: one workflow file, one secret, a token bill that scales with the code reviewed, and a model choice you own. Buy a per-seat product when you want zero operations and a vendor to call — not because the review is better, but because you are buying someone else’s problem instead of running your own. And before you set any of it up, ask whether the review will be read. The harness can make the review happen automatically. It cannot make anyone read it.

Want the same reviewer without running it yourself? OrcaCode Review runs this exact harness as a hosted GitHub App — same open recipe, same per-token bill, no seats.