Qwen3.8-27B Benchmarks: The Full Table, With the Caveats Attached

Qwen3.8-27B scores Terminal-Bench 2.1 at 73.0, SWE-bench Pro at 61.7, LiveCodeBench v6 at 90.3 and OSWorld-Verified at 84.3 — and every one of those numbers is Alibaba's own. The 27-billion-parameter open-weights model hit Hugging Face on August 14, 2026 under Apache 2.0, and as of August 15 nobody outside Alibaba has independently scored its quality. This page is the complete, sourced rundown: all 25 official benchmarks from the model card, what changed versus its predecessor Qwen3.6-27B, what an independent AMD throughput test measured, and which claims you should treat as provisional.

A reading guide before the numbers: "official" here means Alibaba's evaluation printed on the model card at Qwen/Qwen3.8-27B. It is vendor-reported and unreproduced. "Independent" means someone outside the lab measured it — so far that applies to hardware throughput only, not to any quality score. Benchmarks whose harness matters are flagged inline.

The model, one line per spec

• 27B dense parameters (28B counting the vision encoder), 64 layers, hidden size 5120.

• Hybrid attention — 48 Gated DeltaNet linear-attention layers plus 16 full-attention layers, a 3:1 ratio — which is what makes a 262K-token window affordable to run.

• 262,144 tokens native context, extensible to 1,000,000 with YaRN scaling.

• Native image and video input (text output), Apache 2.0 weights, roughly 55.6GB in BF16.

The short version: this is the model meant to take what Alibaba learned building the 2.4-trillion-parameter Qwen3.8-Max and compress it into something a single GPU can run.

The scorecard: every benchmark on the model card

All scores below are Alibaba-reported from the August 14 model card, with the Qwen3.6-27B score the model replaces in parentheses.

Coding. Terminal-Bench 2.1 (agentic terminal coding) 73.0 (63.4); SWE-bench Pro 61.7 (53.5); QwenSWEBench 79.0 (49.3); DeepSWE 1.1 42.2 (13.3); NL2Repo-Bench 42.3 (36.2); LiveCodeBench v6 90.3 (83.9). The SWE-bench Pro and QwenSWEBench runs used the Claude Code harness, per a model-card footnote — worth keeping in mind before comparing them to a model scored on a different harness.

Long-horizon agents and office work. CoWorkBench 70.7 (61.0); JobBench 33.4 (21.8); Agents' Last Exam Pass@1 20.4 / score 42.9 (10.6 / 27.3).

Reasoning and knowledge. GPQA Diamond 89.2 (87.8); Humanity's Last Exam 30.8 (24.0); IFBench instruction-following 79.5 (69.1). HLE is GPT-4o-judged, per the card.

Vision and computer use. OSWorld-Verified 84.3 (63.9); WebArena-Verified 64.8 (48.8); AndroidWorld 81.9 (70.3); MathVision 94.6 with CI / 90.0 without (85.1); BabyVision 85.6 with CI / 65.7 without (28.9); OmniDocBench 1.5 91.1 (89.4); RealWorldQA 85.9 (84.1). "CI" is Alibaba's inference-time enhancement; the gap between its on and off states is the single most informative number on the card about how much score you can buy with extra inference compute.

What actually changed versus Qwen3.6-27B

Every benchmark on the card went up, but the deltas are not uniform — and the shape of the improvement is the story. The gains cluster in agentic coding and computer use, not in knowledge recall:

• DeepSWE 1.1 (agentic coding) 13.3 → 42.2, roughly a 3x jump

• QwenSWEBench 49.3 → 79.0

• Agents' Last Exam score 27.3 → 42.9

• OSWorld-Verified 63.9 → 84.3 and AndroidWorld 70.3 → 81.9

• BabyVision (with CI) 28.9 → 85.6 — the largest relative gain on the card

Meanwhile GPQA Diamond moved 87.8 → 89.2 and RealWorldQA 84.1 → 85.9: real but small. This is not a "smarter at everything" release. It is a release aimed squarely at agents that read screens, use tools, and write code across many steps.

The honest gaps: where it still loses, and the methodology caveats

Against the frontier the picture is flattering but not one-sided. Where Qwen3.8-27B and Claude Opus 4.6 Max overlap, Qwen wins the majority — SWE-bench Pro, QwenSWEBench, CoWorkBench, LiveCodeBench v6, OSWorld-Verified, AndroidWorld, IFBench, and the whole vision block. Against Meta's Muse Glimmer-30B, the natural local-class rival, it wins all eight overlapping benchmarks outright. But it still loses Terminal-Bench 2.1 (73.0 vs 78.2), GPQA Diamond (89.2 vs 91.3), Humanity's Last Exam (30.8 vs 40.0) and NL2Repo-Bench (42.3 vs 47.6) to Claude Opus 4.6 Max. If your workload is the hardest frontier reasoning — frontier math, massively multidisciplinary questions — the 27B is not there yet.

Three caveats before you quote any of this. First, none of it is independently verified; there is no Artificial Analysis index and no LMArena run for the 27B as of August 15, and Alibaba's own harnesses are not the ones third parties will use. Second, the model card uses the Claude Code harness for SWE-bench Pro and QwenSWEBench and a GPT-4o judge for Humanity's Last Exam — comparisons with models scored on other setups will move the numbers. Third, Alibaba's MathVision run used a fixed prompt while competitors got the higher of two variants. None of this means the results are wrong; it means they are a ceiling, not a floor, until someone else runs the model.

What it takes to run it

Qwen3.8-27B is a local model, which changes what "performance" means: throughput on your own hardware instead of a price card. The BF16 weights are roughly 55.6GB; quantized to 4-bit it fits a 24GB card such as an RTX 3090 or 4090. AMD shipped Day-0 support on launch day, and its own llama.cpp/Vulkan measurements — August 2026, three or more runs averaged — put it at 24.5 tokens/s on a Ryzen AI Max+ 395 and 51.8 tokens/s on a Radeon AI PRO R9700 with 32GB, on systems with more than roughly 24GB of variable graphics memory or VRAM. Community tests of the FP8 build on an NVIDIA GH200 held 10 concurrent 262K-context requests with sub-10ms time-to-first-token. Your own numbers will differ — those are someone else's GPUs — but the shape is real: this is the first Qwen release in this class that runs, not just in a review, on a single workstation.

Free weights, or a paid API?

The decision this model forces is the one that divides local from hosted: the weights cost nothing, but your GPUs are not free. Self-hosting means owning the silicon — one 24GB card if you accept 4-bit quantization and slower generation, something bigger if you want the full 262K context at full quality. The hosted alternative on OrcaRouter is $0.33 per million input tokens and $2.40 per million output, with the same 262K context and image-plus-video input, and a p50 time-to-first-token of 225ms measured over the last seven days — no GPU to buy, no VRAM to babysit. The arithmetic is the one you always do with open weights: the token throughput you actually need times your cost per token, versus the flat price of a card. At heavy sustained volume, self-hosting wins; at bursty or modest volume, paying $0.33/$2.40 beats buying silicon you mostly idle.

Bottom line

Qwen3.8-27B is the strongest open-weights model Alibaba has put in this size class, on Alibaba's own numbers: best-in-table at agentic coding, computer use and vision reasoning, with a 262K context window and Apache 2.0 weights. The correct reading of the scorecard is conditional — every figure is vendor-reported, harness-specific, and as yet unreproduced, and the frontier still wins the hardest reasoning benchmarks. If you are deciding whether to self-host it or call it as an API, treat the benchmark table as the promise and the AMD throughput figures as the only independent measurement that exists today. We will update this page the day an independent lab publishes a score.