Qwen3.8-27B on vLLM: Serve It in Production on One or Two GPUs

Yes — Qwen3.8 27B serves in production on vLLM today, and on a single GPU in most cases. The version that counts is vLLM 0.17.0 or newer: it ships the official recipe, the hybrid-attention kernels this model needs, and an O​penAI-compatible /v1 endpoint. For one Blackwell GPU, run the NVFP4 quant — vLLM's own recipe measures it at 24.6 GiB of VRAM at tensor-parallel size 1. For a single 48GB card, run FP8. Full BF16, a 51.7GB checkpoint, wants one 80GB GPU or two 48GB cards in tensor-parallel. The exact commands are below; the first one is a one-liner.

Everything here was verified August 15, 2026, the third day the weights were live. Architecture, context, and license come from the Qwen3.8 27B model card on Hugging Face; the checkpoint size is the sum of the repo's 18 safetensors shards; the vLLM commands and the 24.6 GiB figure come from vLLM's own recipe page for this model. Qwen3.8 27B is Ali​baba's dense 27-billion-parameter multimodal model — Apache 2.0 weights, released August 13–14 — and because the weights are open, you can serve it yourself instead of renting tokens. That fork is what this article is really about.

The facts that matter, with sources

• Architecture — 27B dense (27.8B counting the vision tower and the padded vocabulary), 64 layers, hidden size 5,120, vocabulary 248,320. Official model card, verified today.

• Attention — hybrid: 16 full-attention layers, 48 Gated DeltaNet linear layers in a 3:1 block pattern. Only 16 layers keep a growing key-value cache; the other 48 keep a fixed-size recurrent state instead.

• Context — 262,144 tokens natively, extensible to roughly 1M via YaRN RoPE scaling. Model card, verified today.

• Input — native text, image, and video; text output. vLLM exposes all three through the standard chat-completions API, no separate projector file.

• License — Apache 2.0. This single fact is why the "serve it yourself vs rent the tokens" question exists at all.

• Weights — the BF16 checkpoint totals 51.7GB across 18 safetensors shards (Hugging Face, verified today). Qwe​n also publishes FP8 and NVFP4 checkpoints built for vLLM.

• vLLM requirement — 0.17.0 or newer, with transformers ≥ 5.8.0. vLLM's recipe page, verified today. "Any vLLM" is not a safe instruction; the recurrent-layer kernels are what the new version adds.

• Multi-token prediction — a speculative-decoding draft head ships inside the checkpoint, so you do not need a separate draft model. vLLM documents the flag; no independent speedup figure exists yet.

The GPU ladder — which quant on which card

Three serving formats cover the practical range. Pick by the VRAM you actually have, not by "best quant".

• NVFP4 — 24.6 GiB total (weights plus an FP8 KV cache), per vLLM's recipe at TP1. Fits a single Blackwell-class GPU — in practice a 32GB RTX 5090 or a B200. This is the lowest-latency path and the one that keeps the most context per card: vLLM's recipe reports 6.6M KV-token capacity even at the 1M-context extension.

• FP8 — about 26GB of weights. One 48GB card (L40S, RTX A6000, RTX 6000 Ada) serves it with room for context; two 48GB cards in tensor-parallel give you headroom for longer context or higher concurrency. vLLM's own recipe runs FP8 at TP4 across a four-GPU GB300 tray when you want the largest possible KV cache.

• BF16 — 51.7GB of weights, so a single 80GB GPU (H100, A100 80GB, B200, GB300) or two 48GB cards at TP2. This is the reference-precision option, and it is the one the 1M-context extension command below actually uses.

• MXFP4 — do not use on NVIDIA. vLLM's MXFP4 path is currently missing the linear-method support; the same weights are published as NVFP4, which is the format the NVIDIA recipe actually uses.

Run it — the vLLM commands

The low-latency single-GPU default (NVFP4, one Blackwell GPU), verbatim from vLLM's recipe:

vllm serve Inferact/Qwen3.8-27B-NVFP4 --tensor-parallel-size 1 --max-model-len 262144 --kv-cache-dtype fp8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

The FP8 command from the same recipe (TP4, one GB300 tray, largest KV cache):

vllm serve Qwen/Qwen3.8-27B-FP8 --tensor-parallel-size 4 --max-model-len 262144 --kv-cache-dtype fp8 --reasoning-parser qwen3

For two 48GB cards, keep the FP8 command and set --tensor-parallel-size 2 instead of 4.

Append this to either command to enable MTP speculative decoding:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The 1M-context extension (also from vLLM's recipe):

vllm serve Qwen/Qwen3.8-27B --max-model-len 1010000 --hf-overrides '{"text_config": {"max_position_embeddings": 1010000}}'

Point any O​penAI client at http://localhost:8000/v1 — the endpoint is a drop-in replacement. Two runtime details matter once it is up: thinking is on by default at reasoning_effort xhigh, so turn it off per request with chat_template_kwargs {"enable_thinking": false} or drop it to {"reasoning_effort": "low"} for faster answers. And the checkpoint ships temperature 1.0, top_p 0.95, top_k 20 in its generation config — pass those unless your app already overrides sampling.

What vLLM's own pages do and do not promise

• No 27B throughput numbers yet. As of August 15, vLLM's recipe page publishes no throughput or latency benchmarks for Qwen3.8 27B. The "4,000+ tokens per second per GPU" figure floating around belongs to the Qwen3.8 2.4T-A95B on a 72-GPU GB300 NVL72 rack, is vendor-reported, and is not this model. Community tokens-per-second figures you will see circulate for GGUF under llama.cpp or Ollama — a different runtime and a different workload than vLLM serving.

• The cheap-KV-cache claim is runtime-dependent. The model card says only 16 of 64 layers keep a cache, but that only helps if the serving engine actually implements the Gated DeltaNet layers. vLLM 0.17+ is the version that does; that is why the version pin is the first thing in this article rather than a footnote.

• MTP is built in but unmeasured here. The draft head is in the checkpoint and vLLM documents the flag, but nobody has published an independent speedup figure for this 27B on vLLM yet. Plan to measure it on your own traffic.

• NVIDIA is the tested path. vLLM's recipe is written for NVIDIA GPUs (NVFP4 and FP8). AMD Instinct or Intel Gaudi deployments of this hybrid-attention model are still bleeding-edge, and this article does not pretend otherwise.

Serve it yourself, or rent the tokens

This is where Apache 2.0 does its work. There is no per-token license fee on Qwen3.8 27B, so the only real question is whether you own the hardware or rent the tokens.

• Self-host (this article) — you pay for the GPU once, and every token after that is free. An RTX 5090 you already own makes the NVFP4 command a zero-marginal-cost endpoint with no data leaving the box. If you have to rent the GPU, a cloud 5090 or a pair of A6000s is the line item, and the whole argument only wins if you already have the card or the sustained volume.

• Rent the tokens — because the weights are open, several hosts run it, and the price floor is the hardware cost. Qwen3.8 27B is live on OrcaRouter today at $0.33 per million input tokens and $2.40 per million output — no vendor markup to pass through, since OrcaRouter runs the open weights on its own infrastructure — and the same open weights fund a rate-limited free tier that charges $0 per request and returns HTTP 429 when you pass its cap. A representative 10-million-token month at 70% input runs about $9.51 on the paid tier.

• The decision rule — if you already own the GPU, self-host. If you would have to buy or rent one, the API breaks even fast at side-project volumes, and the same O​penAI-compatible client points at either endpoint, so the code does not change when you move.

When vLLM is the wrong answer

• You are one person on a laptop — vLLM is a serving engine, not a desktop app. For a single-user local run, llama.cpp or Ollama with a Q4 GGUF is simpler and needs a 24GB card, not a Blackwell GPU; our how-to-run-Qwen3.8-27B-locally guide walks that path end to end.

• You need guaranteed throughput with zero ops — self-hosting means you own the paging, the queueing, and the failover. If "the API is down" is not a sentence you want in your vocabulary, rent the tokens instead and let someone else run the fleet.

• You genuinely need the full ~1M context at frontier quality — that is the Qwen3.8 2.4T-A95B's job, served by vLLM or SGLang across a 72-GPU GB300 NVL72 rack. Qwen3.8 27B on one or two GPUs will not match it; our serving article on the 2.4T explains why that model is a different class of problem.

• You are on an older 24GB card — NVFP4 is a Blackwell-format; on Ampere (RTX 3090) or Ada (RTX 4090) 24GB cards, the FP8 path is the vLLM option, and beyond that a GGUF quant under llama.cpp is the pragmatic stop. The same model on a 24GB card is a different article.

• You must serve maximum concurrency on one card — the single-GPU defaults above are the starting point, not the production shape. Tune --max-num-seqs, the KV cache, and the MTP config against your own request mix before you call it done.

Bottom line

Qwen3.8 27B is the rare dense 27B that vLLM serves on a single GPU in production. Update to vLLM 0.17.0+, pull the NVFP4 quant for a 32GB Blackwell card at 24.6 GiB, the FP8 quant for one 48GB card or two in tensor-parallel, and reserve BF16 for an 80GB GPU. The commands are one-liners, the endpoint is O​penAI-compatible, and because the weights are Apache 2.0 you can serve it yourself or rent it at $0.33/$2.40 per million tokens with a free tier — the same client code either way. The one thing nobody has yet is an independent throughput number for the 27B on vLLM, so budget a benchmarking hour after you boot it before you promise anyone a latency number.