Qwen3.8-27B GGUF: Which Quant to Download for Your GPU

Download the unsloth/Qwen3.8-27B-GGUF pack, and match the file to the card you actually own. That is the whole answer: a 24GB GPU (RTX 4090 or 3090) should pull Q4_K_M at 17.1GB; a 16GB GPU should pull IQ4_XS at 15.7GB (or Q3_K_M at 13.8GB if you want context headroom); a 12GB GPU is limited to the 2-bit files, UD-IQ2_XXS at 9.0GB, and that is a compromise to make knowingly. Qwen3.8 27B — Ali​baba's dense 27-billion-parameter model, Apache 2.0 weights released August 13–14, 2026 — runs unusually cheap locally because its hybrid attention caches only 16 of its 64 layers, so a 4-bit quant on 24GB comfortably carries 32K–64K tokens of context. And the GGUF is the only route with zero marginal cost: no API key, no per-token bill, no data leaving your machine.

On sourcing: the architecture, context window, and license are official, from the Qwen3.8 27B model card on Hugging Face, verified August 15, 2026. The GGUF sizes are from the unsloth and ggml-org GGUF repositories, same day. The per-GPU VRAM guidance is unsloth's official recommendation. The KV-cache byte estimates and tokens-per-second figures are community-measured in the first days after release, not vendor specs. Every benchmark named below is Ali​baba-reported and unreproduced.

The file picker, one line per quant

• UD-IQ2_XXS — 9.0GB — the only comfortable fit for a 12GB card; expect visible quality loss.

• UD-Q2_K_XL — 10.7GB — 12GB cards, with almost no context left to spare.

• Q3_K_M — 13.8GB — 16GB cards that want context headroom.

• IQ4_XS — 15.7GB — 16GB cards: the largest quant that fits whole.

• Q4_K_M — 17.1GB — 24GB cards: the sweet spot, and the file this article points you to.

• Q5_K_M — 19.8GB — 24GB, trading context headroom for a small quality gain.

• Q6_K — 22.9GB — fits 24GB if you squeeze; near-lossless.

• Q8_0 — 29.0GB — 32GB, a two-card split, or CPU offload.

• BF16 — 54.7GB — server cards and RAM-heavy CPU setups; reference precision.

Two packs, two Q4_K_M sizes: unsloth's is 17.1GB; ggml-org's is 19.0GB. The unsloth pack uses Dynamic V3.0 (preview) quantization for its UD-* files and is the one most local apps default to; the ggml-org pack ships the standard K-quants plus the separate multi-token-prediction head used for speculative decoding. Either Q4_K_M works — the difference is a few hundred megabytes and the MTP file.

Why this 27B fits on cards smaller than most

Qwen3.8 27B has 64 layers, but only 16 use full attention. The other 48 use Gated DeltaNet linear attention, which keeps a fixed-size recurrent state instead of a growing key-value cache. The model card's own block layout is 3×(Gated DeltaNet → FFN) for every 1×(Gated Attention → FFN) — a 3:1 hybrid ratio. The practical effect: the KV cache is roughly a quarter of what a conventional 27B dense model needs for the same context. Community measurements this week put the cache at about 0.5GB at 8K context, 2.0GB at 32K, and 16.4GB at the full 262K native window. That inverts the usual advice — most of your VRAM budget goes to the weights, not the context, and a 24GB card can run Q4_K_M with 64K–96K context without sweating. You can shrink the cache further with llama.cpp's --cache-type-k flag, which quantizes the cache itself.

Three gotchas that will trip you up on day one

• Old llama.cpp builds reject the file. The GGUF registers the new qwen35 architecture, so you need a current build — anything from before release week refuses to load it with an architecture error. Update llama.cpp first, before you download.

• The template trap. The official Jinja chat template wraps every assistant turn in a think block even when the reasoning trace is empty, which produces nested blocks and truncated history in multi-turn chats — it looks like the model forgot what you said. Run llama.cpp with --jinja, and prefer a pack that ships a corrected chat_template.jinja.

• Vision needs a separate file. Text works out of the box; images and video silently do nothing unless you also load the mmproj projector, roughly 0.9GB. It is not listed on unsloth's GGUF repo page — grab it from the base repository or the ggml-org pack. If you plan to feed documents or screenshots, append --mmproj to the server command.

Run it: the commands

llama.cpp, once you have a current build:

llama-server -m Qwen3.8-27B-Q4_K_M.gguf --ctx-size 32768 --n-gpu-layers 99 --jinja --cache-type-k q8_0

...and add --mmproj Qwen3.8-27B-mmproj-bf16.gguf if you need vision.

Ollama, if you want the library entry: ollama pull qwen3.8:27b, then ollama run qwen3.8:27b. LM Studio and Unsloth Desktop pick up the chat template and the RAM offloading automatically — they are the least fiddly path for a first run.

Local vs API: the honest economics

The GGUF is the free route: zero marginal cost, no rate limits, nothing leaves your machine. It is not the cheap route in absolute terms — you supply the 24GB GPU (a used RTX 3090 or a new RTX 4090, plus electricity), and a 2-bit file on a 12GB card is a compromised experience.

The API route exists because the weights are Apache 2.0, so the marginal cost of serving is near zero and anyone can host. Qwen3.8 27B is live on OrcaRouter today at $0.33 per million input tokens and $2.40 per million output — the provider list price passed through with no markup — and because the weights are open, there is also a rate-limited free tier that charges $0 per request and returns HTTP 429 when you exceed its cap. A representative 10-million-token month at 70% input share runs about $9.51 on the paid tier. If you already own the GPU, local wins on cost; if you do not, renting the tokens beats buying a card for a side project.

When this recommendation is wrong

Q4_K_M on 24GB is the default, not the universal answer. Pick something else when:

• You need maximum quality on a server or workstation — Q8_0 at 29GB or BF16 at 54.7GB with CPU and RAM offload, not a 4-bit file.

• You are on a Blackwell RTX 50-series card — the NVFP4 variant is roughly 1.5× faster in the same 24GB of VRAM and uses an FP8 KV cache for longer context. On a 5090, NVFP4 beats any GGUF on this model.

• You need the full 262K context — budget for a roughly 16GB cache on top of the weights; that pushes a 24GB card down to Q3_K_M, or forces KV quantization.

• You rely on speculative decoding — pick the ggml-org pack, which keeps the multi-token-prediction head; some quants strip it to save about 0.5GB.

• You only have 12GB — an honest answer: a 2-bit 27B is noticeably worse than the same model served properly. Try the free API tier before you commit to a local 2-bit setup; if it is good enough, the GGUF can wait until hardware catches up.

Bottom line

Qwen3.8 27B is the rare 27B that fits a 24GB card without compromise: a 17.1GB Q4_K_M download, a current llama.cpp build, and a KV cache cheap enough that long context costs almost nothing. Download that file if you own the hardware, remember that vision needs the separate mmproj, and expect the template trap the first time a multi-turn conversation looks forgetful. If you do not own the hardware, the same model is a $0.33/$2.40 API with a free rate-limited tier, so there is no reason to run a 2-bit file you are not happy with. The GGUF is the free path; it is just not the only path anymore.