LMArena and the LLM Leaderboards: Three Rulers, Three Different Questions

On 19 February 2026, Artificial Analysis published an article headlined "Gemini 3.1 Pro Preview: The new leader in AI". It led six of ten evaluations and sat four points clear of Claude Opus 4.6. Open that same model's page today and the number is 30, ranked 69th of 200. Google changed nothing. What changed is the ruler: on 19 September 2026 Artificial Analysis revised its Intelligence Index to v4.3.2, retired Terminal-Bench 2.1 in favour of Terminal-Bench 4.0, swapped tau3-Banking for AutomationBench-AA and re-anchored the GDPval-AA Elo scale. A February leader became a September mid-fielder because the measuring instrument moved underneath it.

That single episode is the whole argument for learning to read these boards properly. Claude Opus 5.5 and Claude Fable 5.1, Kimi K3.1 Pro and MiniMax M3.1 Flash, and every model in between are being scored by at least three incompatible things at once, and the searches that bring people here — lmarena, llm ranking, which model is actually best — assume there is one answer. There are three, and they answer different questions.

Three kinds of scoreboard, three different questions

Every number you will see on a leaderboard comes from one of three families, and the family determines what the number can and cannot tell you.

The crowd arena. LMArena shows two anonymous model outputs for one prompt, a person picks a winner, and the votes are aggregated into a rating. It measures preference under blind conditions on prompts nobody chose in advance. Its strength is that no vendor runs the harness and nobody hand-picks the questions. Its weakness is that a voter rewards fluent, confident, well-formatted answers, and a prompt typed by a volunteer is not your workload. A high arena rating means people tend to prefer the output; it does not mean the output is correct.

The composite index. Artificial Analysis runs a fixed suite of evaluations and combines them into a single Intelligence Index, alongside separate measures for speed, latency, price and cost per completed task. This is the most useful of the three for shortlisting, because the harness is the same for every model and the surrounding columns let you weigh quality against money. Its weakness is that a composite embeds editorial choices: which evaluations count, how they are weighted, and what happens when one of them is retired. The v4.3.2 revision is exactly that weakness made visible.

The vendor sheet. Launch material from the lab that built the model. It is the richest source — hundreds of rows, the newest benchmarks, the configurations nobody else has run — and it is the least comparable, because every vendor picks the harness, the effort setting and the opponents. Anthropic's Claude Opus 5.5 table puts its model 29 points ahead of GPT-5.6 Sol on Terminal-Bench 4.0; OpenAI's Sol material leads with Terminal-Bench 2.1, where Sol scores 91.9, and contains no row for Opus 5.5 at all, because Opus 5.5 did not exist when it was written. A table with only favourable rows is not evidence of a sweep. It is evidence of row selection.

The row is a configuration, not a model

The most common misreading of a leaderboard is treating one line as one model. On Artificial Analysis's own LLM leaderboard this week, Claude Opus 5.5 appears three times:

• Claude Opus 5.5 at maximum effort with fallback — Intelligence Index 58, cost per task $5.98, median output speed 98 tokens per second, latency 706 ms, end-to-end 711 ms.

• Claude Opus 5.5 at xhigh effort with fallback — Intelligence Index 56, cost per task $3.46, speed 82 tokens per second, latency 119 ms, end-to-end 125 ms.

• Claude Sonnet 5.5 at maximum effort — 56, $5.46, 141 tokens per second, 427 ms.

Two points of index separate the first two rows. A 42% cut in cost per task and a sixfold latency difference separate them as well. Nothing about the model changed; the effort setting did. Any argument of the form "model X scores higher than model Y" is therefore incomplete until it names the configuration, which is why a single head-to-head number should never settle a procurement decision on its own.

The same rule applies to reasoning effort inside a vendor's own material. Anthropic footnotes that its Claude Opus 5.5 Terminal-Bench run used xhigh effort rather than the default, and that at default medium effort the FrontierCode advantage over GPT-5.6 Sol narrows from a headline gap to seven points. Both numbers are true. Only one of them describes what you will get if you do not set the parameter.

The version trap, which accounts for most bad comparisons

Benchmark names are reused across generations and the numbers do not travel between them. Three examples from this month's coverage are enough to show the shape.

• Terminal-Bench 2.1 versus 4.0 — Google's card for Gemini 3.5 Flash-Lite reports 54.0% on Terminal-Bench 2.1; Artificial Analysis, running 4.0, records 1%. Both are correct, and quoting either without the version is misleading in a way that flatters whichever model you prefer.

• OSWorld-Verified versus OSWorld 2.0 — Anthropic reports 85.0% for Claude Mythos 5 on OSWorld-Verified. Other vendors quote OSWorld 2.0 scores in the sixties. Putting them in one column is not a comparison of models.

• The right model, the wrong version — Claude Mythos 5.1 scores 60.9 on Terminal-Bench 4.0 against Muse Wonder's 52.0, which is the only shared test between those two families. Mythos 5's published 88.0 is Terminal-Bench 2.1. Substituting one generation for the other produces a 36-point gap that does not exist.

The practical habit is small and worth adopting: when a number appears without a benchmark version and an effort setting, treat it as a claim rather than a measurement.

Cost per task is not the price on the card

Leaderboards have quietly added the column that matters most for budgeting, and it routinely contradicts the rate card. MiniMax M3.1 Flash costs $0.15 per million input tokens and $0.60 output; Gemini 3.5 Flash-Lite costs $0.30 and $2.50 — four times more on output. On cost per completed Intelligence Index task, Flash-Lite wins: $0.19 against $0.22.

That inversion is not an error. The metric is tokens actually consumed multiplied by the provider's rates, so it measures how much work each model does to finish the same task. MiniMax M3.1 Flash consumed roughly 180 million output tokens across the index against a median near 100 million for the models Artificial Analysis tracks: it is a verbose reasoner, and verbosity is billed. On tasks where the deliberation buys a correct answer, that is a good trade. On tasks where the answer was never in doubt, it is waste.

This is the column to run against your own traffic before you trust any per-token comparison, because it is the only one that accounts for the fact that models do not spend the same number of tokens arriving at the same place.

What a leaderboard will not tell you

Two blind spots deserve their own heading, because both cost money when ignored.

Abstention. Artificial Analysis's AA-Omniscience evaluation scores MiniMax M3.1 Flash at 16: 56% of factual questions answered correctly, and a 93% hallucination rate. That second figure does not mean 93% of answers were wrong — it means that of the responses that were not correct, 93% were outright wrong rather than partial answers or refusals. The model usually answers when it should decline. Gemini 3.5 Flash-Lite carries no AA-Omniscience score at all, which is missing data, not a clean bill of health.

Availability. A score is a statement about a moment. Gemini 3.1 Pro has spent seven months as a preview with no GA date, disappeared from the AI Studio model selector in mid-September with no vendor explanation, and still carries a ranking. Grok 4.7 was a day old when it was scored 46 on the index. One of those is a number about a model you can call next month; the other is a number about a model that may not exist next quarter. The leaderboard does not distinguish between them.

Where our own board sits, and why it is a fourth thing

OrcaRouter runs an AI model leaderboard that deliberately does not reproduce any of the above. It ranks on what our own gateway observes — production traffic and community Battle Mode votes — and blends that with LMArena votes, independent benchmarks and ecosystem adoption under fixed, published weights. It is updated daily, lists models within 48 hours of arrival, and splits into Text, Coding, Vision, Math, Reasoning and Agentic views rather than pretending one ordering serves every job.

That gives it a different failure mode from the other three, which is worth stating plainly: it can only rank models we route, so its blind spots are commercial rather than methodological. What it adds is the one signal none of the others have — whether the thing actually answered, at what speed, for what blended price, on traffic that was not constructed for a benchmark.

Reading one in five minutes

• Find the benchmark version and the effort setting before you read the score. If either is missing, the number is a claim.

• Read the cost-per-task column, not the per-token price. They disagree more often than they agree.

• Treat a composite index as a shortlist, not a verdict. It is a reweighting, and reweightings get revised — the same revision that demoted Gemini 3.1 Pro demoted dozens of models nobody had touched.

• Separate capability gaps from score gaps. A 65,536-token output ceiling or a model that cannot read video is a fact about your workload; two points of index is not.

• Check when the model shipped and whether it is still a preview. A ranking without an availability date is a ranking of something that might not be there when you deploy.

The leaderboard is the beginning of the question, not the answer to it. The only measurement that settles a choice is your own traffic on your own prompts — which is why the useful thing a gateway can offer is not another ranking but the ability to run the two candidates side by side on one key and compare the invoices.

Sourcing note: the Gemini 3.1 Pro February-to-September movement, the v4.3.2 revision contents, the three Claude Opus 5.5 rows, the AA-Omniscience figures, the cost-per-task values and the Intelligence Index scores are Artificial Analysis's, read on 26 October 2026. The Terminal-Bench 2.1 and 4.0 figures, the OSWorld-Verified and OSWorld 2.0 figures, the 60.9 and 52.0 Terminal-Bench 4.0 scores, the MiniMax token-consumption estimate and the vendor-sheet comparisons are from Anthropic's, Google's, MiniMax's and OpenAI's own published material respectively, and are vendor-reported unless attributed above to Artificial Analysis. OrcaRouter's leaderboard description, weights and update cadence are our own published methodology.