Gemini 4 Carbon Benchmark: What 60 on AA v5.0.1 Actually Measures

A benchmark score is only useful if you know what the harness asked, how long the model was allowed to think, and what version of the test produced the number. For Gemini 4 Carbon the answers are: nine agentic and scientific evaluations, as long as it wanted, and Intelligence Index v5.0.1 — the current version, which is why the 60 is quotable and most numbers you will find next to it are not. The Gemini 4 Carbon API page lists the price and the endpoint; this piece is about the score.

What was actually run

AA-Briefcase v1.1, AutomationBench-AA, Terminal-Bench 4.0, FrontierCode-AA 2.0, AA-Omniscience v1.1, GDP.pdf, HLE-Diamond, CritPt v2 Hard Subset, and Terminal-Bench-Science 0.1. Read that list before you generalise from it. Three of the nine are terminal or automation work, two are document-grounded reasoning over long inputs, two are hard science and mathematics, and one is frontier code. There is no web-agent evaluation and no multi-turn customer support simulation in the set, so a great score says nothing about those.

The score

Carbon posts 60, which is rank 4 of 78 on v5.0.1. The leader is Claude Fable 5.2 at 65. The gap of five points is meaningful but not categorical: both models clear the same nine evaluations, and the cheapest way to read the difference is as roughly one additional hard task solved per twenty attempted at Carbon's level of competence. Treat that as a direction, not a rate.

The version guard matters more than the rank. Models scored on v4.3 or v4.3.2 — including most of the names you will see quoted in comparison posts — are not on the same scale and cannot be placed in the same table. If a chart mixes them, the chart is wrong regardless of who published it.

The practical consequence is that you cannot compare Carbon to a model you evaluated yourself last quarter unless that model has also been re-scored. Either re-run your own harness on both, or restrict the claim to the v5.0.1 cohort and say so in the sentence where you quote it.

The time cost

Time to first token is 64.7 seconds, rank 69 of 77 models with latency data. Total response time is 28.4 minutes per task across the nine evaluations. Once Carbon begins emitting, it runs at 72 tokens per second, which is ordinary. The wall clock is dominated by thinking, not by writing.

That is a design choice rather than a defect, and it is worth being precise about what it implies. A harness that lets the model reason until it is satisfied will produce a slow but strong score; the same model behind a gateway on live traffic reports a p50 of 6.89 seconds to first token and a p95 of 10.00 seconds, because most real requests are not HLE-Diamond. The two figures measure different things and both belong in a capacity plan.

The money cost

Running the index against Carbon cost $5.48 at $2.00 per million input tokens and $10.00 per million output, for a blended $2.70 per million. AA also reports 205 million output tokens for the run with a verbosity figure of 2.69 percent. The token count, not the percentage, is the number to carry forward: it tells you that a nine-task agentic evaluation consumed the better part of a quarter billion output tokens.

For comparison, the same index against Claude Fable 5.2 cost $26.54 — 4.8 times as much — at $8.00 in and $40.00 out. That is the correct framing for the pair: the five-point score gap is bought at nearly five times the token spend.

Put as a ratio, Carbon pays $0.09 of benchmark spend per index point and Fable 5.2 pays $0.41. That is not an argument that the cheaper model is better — the point of a frontier model is the tasks the cheaper one fails — but it is the argument for routing by difficulty instead of picking one model for everything. The 90 percent of calls that a 60-point model clears do not need a 65-point one.

The other ratio worth carrying is time. Carbon spends 28.4 minutes per task and Fable 5.2 spends 46.8, so the more expensive model is also 1.6 times slower end to end. On an asynchronous queue that is a throughput constraint; on a synchronous call it is the difference between a spinner and a timeout. The Gemini 4 Carbon release notes cover what shipped with the model, and if the question is whether it supersedes the previous Gemini generation, Gemini 4 Carbon vs Gemini 3.1 Pro is the direct comparison.

What the benchmark does not cover

There is no cyber or vulnerability-detection harness in this set. If you are evaluating Carbon for security work — scanning, triage, patch generation — these nine evaluations tell you about its reasoning and nothing about its recall on real vulnerabilities. The same is true of long-horizon coding on a specific repository, which FrontierCode-AA approximates but does not replace.

There is also no measurement here of what happens under load. The live error rate on OrcaRouter traffic for Carbon was 39.7 percent over the last seven days, against 10.8 percent for Fable 5.2. A benchmark harness retries and absorbs that; your application has to decide how.

Reproducing any of this on your own traffic

You cannot reproduce AA's 64.7 seconds without AA's harness, and you should not try: the useful thing is to measure your own p50 and p95 and compare them to the gateway's published 6.89 and 10.00 seconds, because that tells you whether your prompts are ordinary or unusually hard. Log time to first token, tokens out, and the non-2xx count per route. If your error rate is above the 39.7 percent the gateway reports across all providers, the problem is your side of the request.

How to use this in a decision

If your work resembles the nine evaluations — terminal tasks, automation, documents with long contexts, hard science — Carbon at 60 and $2.70 blended is the value pick of the current v5.0.1 cohort, and the 64.7-second think time is a cost you can absorb by making the call asynchronous. If your work is interactive, the same model with a p50 near seven seconds through the gateway is a different proposition entirely, and the honest test is your own traffic rather than AA's harness.

For the head-to-head against the model that beat it, Gemini 4 Carbon vs Claude Fable 5.1 puts the pair side by side, and the LLM API pricing comparison covers where $2.70 blended sits against the rest of the catalogue. Re-check the version on any chart you reuse: v5.0.1 is the only scale this 60 belongs to.