Qwen 4 Leak: SGLang's Host-Staging PR Shows How the 47.7 GiB PLE Table Fits on One GPU

The number that will decide whether Qwen 4 is a model you can serve yourself is not its parameter count. It is 47.7 GiB — the size of the n-gram embedding table that rides alongside the Qwen4 architecture, separate from the weights, and has to live somewhere while the model decodes. A pull request opened in the SGLang repository on 18 September 2026, titled [Qwen4-Exp] Add host staging for file-backed PLE, is an attempt to stop that table from dictating how much RAM a machine needs before it can run at all. Qwen 4 is still unreleased: no model card, no weights, no catalog entry, no date. The only shipped model that instantiates this architecture is Qwen3.8-Flash-Next, the open-weight preview published on 26 August 2026, whose configuration declares model_type=qwen4_exp — the same string that gives the pull request its name. Its production sibling Qwen3.8-Flash is the version an API caller can actually reach today. Everything in this piece about Qwen 4 is inference from that preview and from engine code; the pull request is open and unmerged, so read all of it as a signal, not a shipped capability.

What the signal actually is

Pull request sgl-project/sglang#40235 is not a launch and it is not merged. It sits on one commit, opened by contributor Dev-Jahn, merging a branch called task/ple-host-staged into SGLang's main line. Nine code owners have been requested for review and all of them show as awaiting, so at least one approving review stands between this and main. Three CI jobs — the base PR test, the extra PR test, and the AMD ROCm run — are failing on the open commit. That is the normal condition of a large engine change in flight, and it is also why the interesting part of this PR is not whether it lands but what its author had to measure to argue for it. The description carries roughly 1,400 added lines including tests, and a benchmark table that is the most concrete public data anyone has published about running this architecture on a single GPU.

Why the table is the whole story

Per-Layer Embeddings are the structural oddity of this generation. Where a normal model puts one token embedding at the front, the Qwen4 design carries a large n-gram table — bigrams and trigrams, hashed into a vocabulary far larger than the tokenizer's — and feeds per-layer embedding lookups from it throughout the stack. Alibaba's own preview material describes the n-gram component as tens of billions of parameters on top of the 125B-parameter MoE body; community teardowns of the released checkpoint put the table file at 47.7 GiB. Those figures come from vendor and community sources rather than independent reproduction, and the exact relationship between the preview's table and whatever Qwen 4 ships is unknown.

What is not in doubt is the engineering consequence. A 47.7 GiB side table that must be consulted on every decode step is not something you can quietly push into a corner of VRAM. On a 96 GB card it competes directly with the KV cache; on smaller cards it simply does not fit. That is why SGLang, which stood up day-0 support for the preview in late August, has spent the following three weeks producing one pull request after another about this single data structure rather than about the model around it.

What was broken before this PR

SGLang already had two ways to hold the table, and both had a hard edge.

• Pinned keeps the whole table in host RAM and reads it from there. It works, it is fast, and it makes the host memory requirement absolute — there is no smaller version of it.

• File-backed, added earlier in a separate pull request, keeps the table in a sparse file and lets the gather kernel read the mapping directly, so the operating system's page cache decides how much of it is resident. The catch is a hardware one: that direct-read path requires the GPU to report cudaDevAttrPageableMemoryAccessUsesHostPageTables — the capability the PR's own text glosses as "the GB10 class". On a GPU without it, the file backend is refused outright and pinned is the only option left.

The gap that creates is not theoretical. A separate report filed against the same code path documents a user on two RTX 3090s whose per-rank share of the table came to 23.84 GiB against 23.56 GiB of usable memory — 0.28 GiB short, with 188 GiB of host RAM sitting free. In that configuration the file backend was rejected by the hardware check and the ordinary CPU-offload flag raises when combined with the PLE offload flag. Being three hundred megabytes short with a hundred and eighty gigabytes spare is precisely the shape of problem this pull request exists to remove.

What host staging changes

The mechanism the PR adds is a staging layer between the file and the device. Instead of asking the GPU to dereference host pages, a CPU-side component reads the rows it needs through the loader's existing mapping and advises the kernel to pull those pages in ahead of time; an environment variable, SGLANG_QWEN4_PLE_FILE_PREFETCH, turns the advice off if you want to measure without it. Each PLE layer then gets two pinned buffers of 8,192 rows — about 1.25 MiB each at FP8 — and one worker. Rows are gathered into one buffer while the other is being copied to the device, so the gather and the transfer overlap instead of serialising. N-gram identifiers are hashed on the host rather than the device. Graph replay gets a preparation call before each replay, and the launching thread waits on the previous step.

That last detail is the cost, and the PR states it plainly: roughly 0.5 to 1 millisecond added per decode step over the pinned path. Everything else is the payoff. Measured on a single RTX PRO 6000 Blackwell with 96 GB, an AMD EPYC host with 377 GiB of RAM, CUDA 13.2, using the public FP8 and NVFP4 checkpoints of Qwen3.8-Flash-Next:

• Host page cache, FP8 TP4/EP4 — 71 GB pinned and uncapped, versus 49 GB at a 64 GB cap, 15 GB at 32 GB, and 6 GB at a 24 GB cap

• Decode latency, same runs — 8.62 ms per token at concurrency 1 pinned, versus 9.16 / 9.20 / 9.12 ms across the three capped file runs

• Concurrency 16 — 16.54 ms pinned versus 17.62 / 17.85 / 17.37 ms capped, which is 893 tokens per second down to 827–840

• Prefill throughput — 620 tokens per second at 8k pinned versus 624 / 630 / 631 capped; at 32k, 1,347 versus 1,358 / 1,359 / 1,361

• NVFP4 TP2 — 69 GB pinned versus 24 GB with the file backend at a 32 GB cap, at 8.87 ms versus 9.18 ms

• Single-GPU NVFP4 — 69 GB pinned versus 51 GB at a 64 GB cap, at 6.44 ms versus 6.73 ms

• The alternative it beats — reading the same file through host memory management on that GPU gave 10.8 ms at concurrency 1 and 46.5 ms at concurrency 16, which the PR describes as 2.8× the pinned latency at that concurrency

The accuracy side is reported as clean. Deterministic greedy output over eight prompts of 256 tokens was token-identical between the pinned and file paths on FP8 TP4, and GSM8K came out at 97.6% pinned against 98.0% file at the 64 GB cap — a six-question gap the author attributes to run-to-run variation rather than to the offload path. All of these numbers are the pull request author's own, measured once, on one machine, and nobody has reproduced them.

The costs the PR admits

A fair reading of this pull request includes what it refuses to do. Several execution modes are rejected at construction time rather than silently degraded, and each rejection names the pinned backend as the fallback: prefill CUDA graphs, data-parallel attention, the prefill-decode multiplexing path, two-batch overlap, the DLLM decode graphs, and compact ragged verify graphs are all out. Just as importantly, it adds no new flag and no new user-facing switch — the staging path is what the file backend does on hardware that previously could not use it at all. And the accuracy run carries a caveat the author volunteers: the capped runs never held the entire table, because the table is 47.7 GiB and the caps go as low as 24 GB, so a workload with a genuinely flat, unpredictable access pattern over the whole table is not what was measured.

Why this matters for Qwen 4 specifically

Strip away the engine detail and the pattern is legible. Alibaba shipped an architecture preview on 26 August with instructions for the open-source community to prepare runtimes, quantization and inference engines ahead of the full family. SGLang did that, and then spent three weeks filing pull requests about the one component that makes the architecture awkward to deploy. Read as a forecast, that is a statement about what Qwen 4 will need from your hardware, not about what it can do on a benchmark.

It also sharpens the timeline question. Qwen 4 has not been released, and the September speculation around it points at Alibaba's Apsara Conference on 22–24 September in Hangzhou — the venue where previous Qwen generations were announced. Nothing about that is confirmed, and the pattern from the last preview cycle is that an architecture preview precedes the full family by months rather than weeks. A pull request opening three days before that conference is suggestive and nothing more.

The honest summary of where this leaves a reader: Qwen 4 does not exist, Qwen3.8-Flash-Next does, and the second one tells you what the first will cost you to run. If the 47.7 GiB table keeps shrinking in effective footprint — and three weeks of pull requests say it is being worked on hard — then the deployment bar for the Qwen 4 family is lower than the preview's launch week suggested.

What you can actually do with this today

Nothing in this pull request is available on main, and the model it targets is not something you can call through an API. The open-weight Qwen3.8-Flash-Next checkpoint is a self-hosting story: you pull the weights, you serve them yourself, and the file-backed PLE path is what you are reading about. It is not routed here. What is routed is the production variant — Qwen3.8-Flash, reachable as qwen/qwen3.8-flash at $0.15 per million input tokens and $0.47 per million output, with a 1M-token context and text, image and video input — and the larger Qwen3.8-Max at $2.00 and $6.00.

That distinction is the useful one for a reader deciding what to do this week. The preview is research you run yourself; the served variant is the same architecture's production path, and it is one endpoint away. OrcaRouter passes provider list price through at 0% markup, so a vendor price change on that endpoint shows up the same day rather than at the next billing cycle, and every model on the key is reachable through one OpenAI-compatible base URL instead of a separate contract, SDK and credential per vendor. For an architecture this young — where the engine work is still landing weekly and the roadmap has not been published — automatic failover across providers is the practical way to depend on the served variant without betting a production path on a single provider's uptime. All of that is about the models you can call today. It says nothing about Qwen 4, which is not one of them.

What we still don't know

Whether Qwen 4 will ship the same table at the same size. Whether the pull request merges at all — it has three failing CI runs and no reviews yet. Whether the 0.5 to 1 millisecond per-step penalty holds up outside the author's low-concurrency configurations. And whether Alibaba says anything at Apsara on 22 September. On the current evidence, the safest thing to conclude about Qwen 4 is not what it scores, but how much machinery the industry is building just to make it fit — which is itself a useful thing to know before the model has a name in any catalog.