Nanbeige4.3-33B vs Qwen3.8 27B: Auditing the 18-of-20 Claim

Nanbeige4.3-33B beats Qwen3.8-27B on 18 of 20 benchmarks. That is the claim travelling around the internet this week, and it is technically what the model card says. It is also a sentence with a denominator, and the denominator is the interesting part. Of those 20 rows, 19 are point-scored and one is a dollar figure; four of the 18 wins are by less than a single point; and at least three of the Qwen numbers appear to be copied from Qwen's own published results rather than re-run by Nanbeige. None of that makes the claim false. It makes it narrower than it sounds.

The matchup is unusual because Nanbeige chose Qwen3.8-27B as its sparring partner. Qwen3.8-27B shipped in mid-August 2026, is 27.78 billion parameters dense, Apache 2.0, multimodal on input, and — unlike Nanbeige4.3-33B — has an independent Artificial Analysis Intelligence Index of 34. So this is the one comparison in the batch where the opponent has a verified score and the subject does not. That asymmetry is the best tool we have for auditing the table.

Who wrote the claim, and what the denominator contains

The table is Nanbeige's. It was published on the Hugging Face model card of Nanbeige4.3-33B, in a repository created on October 24, 2026. There is no accompanying blog post, no launch event, and no third-party reproduction. Every number in the Qwen column, and every number in the Nanbeige column, was produced or selected by the lab making the comparison.

The 20 rows cover three groups: agentic coding (Terminal-Bench 2.1 and 4.0, DeepSWE, SWE-Bench Pro, SWE-Atlas-QnA, NL2Repo, Scicode, SkillsBench), agentic cowork (Automation Bench, JobBench, GDPval, Claw-Eval, Toolathlon-Verified, DeepPlanning, EnterpriseOps-Gym, YC-bench, Tau3-Bench), and reasoning (GPQA Diamond, AIME 2026, AA-LCR). The harness is disclosed more carefully than most vendor cards: Terminus 2 for Terminal-Bench 2.1, mini-swe-agent v2.4.6 for Terminal-Bench 4.0, DeepSWE and SWE-Atlas-QnA, Claude Code for NL2Repo and JobBench, and GPT-6 Sol as judge for SWE-Atlas-QnA. That disclosure is to Nanbeige's credit. It is also the reason the comparison is harder to read than a single win count.

The two rows Nanbeige loses

• SWE-Bench Pro — Qwen3.8-27B 61.7 against Nanbeige4.3-33B 60.3. A 1.4-point Qwen lead.

• GDPval — Qwen3.8-27B 86.4 against Nanbeige4.3-33B 84.8. A 1.6-point Qwen lead.

These are the two rows the "18 of 20" headline quietly rounds away, and they point in a consistent direction. Both are benchmarks of professional, economically valuable work: SWE-Bench Pro is the harder software-engineering variant, and GDPval measures output quality on real occupational tasks. Qwen wins both by margins that are small in absolute terms but stable across two different task families. If you are choosing between these models for professional work rather than for a leaderboard, those are two of the rows you should weight most heavily, and they are the two where the subject loses.

Nanbeige's card does not explain the losses. It does not need to — a vendor is entitled to publish a table it wins. But a reader should notice that the losses cluster exactly where the benchmark is closest to "would a professional accept this output," and the wins cluster where the benchmark is closest to "did the agent complete the loop."

Wins that are rounding errors

Four of the 18 wins are decided by less than one point.

• Tau3-Bench — 48.8 vs 48.5. Three-tenths of a point.

• GPQA Diamond — 89.5 vs 89.2. Three-tenths.

• AIME 2026 — 95.2 vs 94.5. Seven-tenths.

• EnterpriseOps-Gym — 41.0 vs 40.2. Eight-tenths.

Two more are within about two points: Toolathlon-Verified at 70.4 vs 68.5, and AA-LCR at 84.0 vs 82.0. On benchmarks of this size and variance, a sub-point margin is not evidence of superiority. It is evidence that two models are in the same band. A fair restatement of the headline would be "Nanbeige leads clearly on 12 of 20, ties on 6, and trails on 2" — which is still a strong result, and a more useful one.

The large wins are where the real argument lives. DeepSWE at 65.5 vs 42.5 is a 23-point gap. Terminal-Bench 4.0 at 23.8 vs 6.3 is 17.5. Automation Bench at 54.2 vs 38.3 is 15.9. DeepPlanning at 63.3 vs 49.2 is 14.1. YC-bench, a dollar-denominated task, reports $1,025,437 vs $641,290. Those are not rounding. They say Nanbeige is better at long-horizon, multi-step execution — the kind of work where a model has to keep a plan alive across dozens of tool calls. That is a coherent story, and it is consistent with the model's 512K native context and its agent-oriented recommended settings.

The columns may not be the same experiment

Here is the observation that complicates the whole table. Three Qwen figures match Qwen's own published numbers exactly: Terminal-Bench 2.1 at 73.0, SWE-Bench Pro at 61.7, and GPQA Diamond at 89.2. Qwen publishes all three. The rows Qwen does not publish — DeepSWE, SWE-Atlas-QnA, NL2Repo, JobBench, Automation Bench, DeepPlanning, YC-bench, Tau3, EnterpriseOps-Gym, Claw-Eval — are precisely the rows where Nanbeige's margins are largest.

That pattern is consistent with two different methods inside one table: copied numbers for benchmarks the opponent already published, fresh runs for benchmarks it did not. It is an inference, not a disclosed fact — Nanbeige's card says all models were evaluated at the highest thinking effort with the same inference parameters and harness, which would argue against copying. But the exact match on three independent figures is a coincidence worth flagging, and if the Qwen column is a mixture, then "18 of 20" compares Nanbeige's runs against Qwen's runs on some rows and Nanbeige's runs against Nanbeige's runs on others.

There is a second, subtler problem. Nanbeige's own model is being run in the harness its card recommends: XML tool calls, preserve_thinking=true, temperature 1.0 for agentic work. The card does not say whether Qwen3.8-27B was given its own recommended settings, or dropped into Nanbeige's. If the latter, part of the 23-point DeepSWE gap is a configuration difference rather than a capability difference. This is the same caveat that applies to every vendor comparison, and it is not a reason to dismiss the result — it is a reason to require a neutral re-run before treating it as settled.

The independent score only one model has

Artificial Analysis scores Qwen3.8-27B at 34 on its Intelligence Index, measured at the xhigh reasoning configuration, with 45.2 output tokens per second. Latency varies by provider: our own page reports a 2.95-second p50 time to first token, against the 3.8 seconds Artificial Analysis measured on the endpoint it sampled. That index is a composite of ten evaluations, including Terminal-Bench 4.0, SciCode, AA-LCR, GDPval-AA and AutomationBench-AA — several of the same benchmark families Nanbeige reports on its own.

Nanbeige4.3-33B has no Artificial Analysis entry. That is not a scandal; the model is two days old. But it means the two sides of this comparison are not epistemically equal. Qwen's 34 is a number someone else measured. Nanbeige's 80.9 on Terminal-Bench 2.1 is a number Nanbeige measured. If both are accurate, Nanbeige would likely land well above 34 on the same composite — its Terminal-Bench 4.0 of 23.8 and SciCode of 52.6 are strong rows. The point is that "likely" is doing the work, and it will keep doing it until an independent evaluator runs the model.

This is also the practical reason to treat the 18-of-20 claim as an opening argument. The claim is falsifiable in a specific, cheap way: submit Nanbeige4.3-33B to an independent evaluator and compare the composite. Until that happens, the strongest honest statement is that Nanbeige claims a lead on long-horizon agentic work and trails slightly on professional-output quality, and that only one of those two models has been checked by anyone other than its maker.

Cost per task, not per token

Qwen3.8-27B is the only one of these two you can call today. On OrcaRouter it is $0.33 per million input tokens and $2.40 per million output — below the $0.50 and $3.00 Artificial Analysis lists for Alibaba's first-party endpoint. This one is served on our own infrastructure rather than resold at provider list, which is why it lands under the first-party price; on models we do pass through, the 0% markup rule means the provider's list price is what you pay. Either way, per-token prices vary enough by route that the first-party endpoint is not automatically the cheap one. The model also accepts images and video alongside text, and carries a 262,144-token window.

The per-token price is the wrong unit for this comparison, though. Reviewers report that Qwen3.8-27B is unusually token-hungry — roughly three times the tokens of Qwen3.6-27B on comparable tasks — and Artificial Analysis estimates about $1.01 per Intelligence Index task at the xhigh setting. A model that wins on price per token and loses on tokens per task can still be the more expensive choice. Nanbeige4.3-33B has no per-token price at all: the weights are free, no inference provider deploys it, and running it means a four-GPU tensor-parallel deployment held open whether traffic arrives or not.

That is where a routing layer earns its place. Qwen3.8-27B is reachable on our key today; Nanbeige is not, and we will not pretend it is. What you can do is keep Qwen as the production path and route a slice of real traffic to a self-hosted Nanbeige endpoint to measure task completion and tokens consumed, rather than trusting a table. Automatic failover matters here: an experiment endpoint that dies mid-run should degrade to the incumbent, not take the agent down with it. The decision then rests on your own cost-per-completed-task, which is the only number that combines both models' token behaviour honestly.

What would falsify this

Three results would settle the question, and none of them requires a new model.

• An Artificial Analysis entry for Nanbeige4.3-33B. If it lands near or above 34, the vendor table's direction is confirmed and the magnitude becomes debatable. If it lands well below, the 18-of-20 claim needs a much better explanation than harness choice.

• A neutral re-run of DeepSWE and Terminal-Bench 4.0 with both models in their own recommended configurations. That is the specific experiment the current table does not perform, and it targets the rows carrying most of Nanbeige's margin.

• A published tokens-per-completed-task figure for both. Per-token prices cannot compare a dense 33B you host against a token-hungry 27B you rent. Cost per finished job can.

The verdict we would defend

Nanbeige4.3-33B probably is the stronger long-horizon agent model of the two. The margins on DeepSWE, Terminal-Bench 4.0, Automation Bench and DeepPlanning are too large and too consistent in direction to be entirely harness artefact, and the 512K native context fits that story. Qwen3.8-27B probably is the stronger model for professional deliverable quality, where it wins both of the rows that measure it, and it is unambiguously the more usable one: independent score, hosted API, multimodal input, and a price you can put in a spreadsheet today.

"18 of 20" is a real result dressed as a bigger one. The accurate version is that a two-day-old model from a lab nobody has independently evaluated claims a clear lead on agentic execution, a tie on general reasoning, and a small deficit on professional output — against an opponent that has been out for ten weeks and has been measured by someone other than its maker. That is a genuinely interesting claim. It is not yet a reason to switch.