Qwen 4: The vLLM 'qwen4 fuse op' PR Named the Architecture — Apsara 2026 Named the Family

A pull request titled "Add qwen4 fuse op" has been sitting open in the vLLM repository since August 26, and for most of that month it was the closest thing to a description of Aliba​ba's next generation of models that existed outside the company. That changed on September 22, 2026, when Aliba​ba's Qwe​n lead Liu Da Yiheng used the opening of the Apsara conference in Hangzhou to name the family: Qwe​n 4 Max, Qwe​n 4 Flash, Qwe​n 4 Plus, and Qwe​n 4 27B — the open-weight tier, and the one the local-inference world has been waiting on. Two days later, with the conference closed, the company put that family on a roadmap for the first time: Qwe​n 4 in training, and a successor line — Qwe​n 4.5 and Qwe​n 5 — projected at 5 to 10 trillion parameters, a figure Aliba​ba Group CEO Eddie Wu gave on stage and the company then repeated in its own English release. What the roadmap is not, two weeks on, is a schedule. The timing that now circulates is a window — October, or early November — and none of it traces back to anything Aliba​ba has published. The company's own written material carries no month, no quarter and no date; the model has no identifier, no context window and no price; and no Qwe​n 4 repository exists in the Qwe​n organisation on Hugging Face. Treat the window as a wish rather than a plan. The pull request — vllm-project/vllm#53909, still open, still unmerged and still awaiting review — carries no release date and no parameter count either. It adds standalone Triton kernels for three operator families: HyperConnection, Qwe​n Sparse Attention (QSA), and PLE state operations. Its title says "qwen4," and its operator list is a near-exact match for the architecture Aliba​ba shipped in the open weights of Qwen3.8-Flash-Next — a multimodal mixture-of-experts model, released on August 26, 2026, whose Hugging Face card reads "A Preview of the Qwen4 Architecture." Qwe​n 4 itself is still unreleased. The family has a name, a slot on a roadmap and a rumored window; the kernels are still the only thing that describes it.

This is a what-we-know-so-far piece, not a launch story. Aliba​ba has now named the family on a public stage and placed it on a company roadmap, but naming is not shipping: there is no model card, no weights and no API for any Qwe​n 4 tier, and every architecture claim below still comes from the Qwen3.8-Flash-Next preview and the engineering artifacts around it rather than from any Qwe​n 4 document. The hard evidence is the vLLM pull request still under review, the released preview model, an sglang PR filed on August 27 that adds native support for the same Qwe​n4-Exp architecture config, and the qwen4_exp entry that has since landed in vLLM's own model registry on main — Qwen4ExpForCausalLM, Qwen4ExpForConditionalGeneration and Qwen4ExpMTP — alongside the Rust-frontend multimodal work, which merged on September 1. The conclusion we draw from them — that the operators in the PRs are the architecture Aliba​ba previewed — is a reading of that evidence, and every specific below is labeled accordingly. One claim here is not Alibaba's at all: the October-to-early-November window comes from a watchlist post on X and the tracker pages that repeated it over the last fortnight, it is unreproduced, and it is carried as community-sourced wherever it appears.

What Alibaba actually said on September 22

Aliba​ba previewed the Qwe​n 4 family at the opening of its Apsara (Yunqi) conference, which ran in Hangzhou from September 22 to September 24, 2026, with Qwe​n lead Liu Da Yiheng on stage. Four tiers were named at the opening, on the structure the 3.8 generation already established:

• Qwe​n 4 Max — the flagship tier, and the one positioned against the top model from every other frontier lab.

• Qwe​n 4 Flash — the high-throughput tier, aimed at latency-sensitive, high-volume work rather than peak reasoning.

• Qwe​n 4 Plus — the balanced middle tier.

• Qwe​n 4 27B — the open-weight tier: the size class Qwe​n has shipped open weights in before, and the one written about as the local-running option.

The 27B entry is the concrete part of the announcement, and it is what the sourcing signal behind this piece was reacting to. In the 3.8 generation the 27B tier was the open-weights drop that was converted to MLX, quantised to NVFP4 and fine-tuned within days of release, so a named Qwe​n 4 27B gives that downstream ecosystem a target again. It is a target and not a release, and the distinction is worth holding onto: Aliba​ba has published no parameter breakdown, no licence, no context window and no date for it, and the tier name does not even say whether it is dense or mixture-of-experts. Everything that made the previous 27B tier runnable — the size class, the single-machine focus — is inherited from the product line's history, not specified for this model.

What the conference did not produce is the thing a reader can plan against. Aliba​ba's stated position — vendor framing, delivered on stage and reported by multiple independent outlets rather than independently verified — is that Qwe​n 4 is built on a new-generation architecture, that it is in training, and that it arrives "soon." No tier has a context window, a price, a modality list or a published score. The only dated item at the conference was a next-generation video model slated for November. One number from the same stage needs keeping in its own lane: the 5-to-10-trillion-parameter ambition belongs to the Qwe​n 4.5 and Qwe​n 5 timeframe, not to Qwe​n 4 — no parameter count was attached to any Qwe​n 4 tier. That figure is no longer a leak or a circulating claim: it is in Aliba​ba's own English release, attributed to CEO Eddie Wu's keynote, and it is stated as a projection with no date and no architecture behind it — company-stated, and still unreproduced.

So the naming changed the status of the family, not the state of the evidence. Everything below is still the serving layer's account of what this architecture is — and with the announcement adding no specifications of its own, that account remains more detailed than anything Aliba​ba has published.

What the "qwen4 fuse op" PR actually is

Pull request #53909 in vllm-project/vllm is open, from a contributor fork, and still waiting on code-owner review — automated review was disabled precisely because it came from a fork. It is small, kernel-shaped work:

• Three HyperConnection kernels — a grouped Ge​ma RMSNorm, a gate reduce, and an injection combine.

• Three QSA kernels plus three vendor-baseline wrappers for comparison.

• PLE state gather and masked-scatter kernels.

• Thirteen passing tests covering dtype and shape guards, paging metadata, cache slots, MRoPE layout, strided state, null and duplicate indices, and deterministic execution.

Two details matter beyond the op list. First, the implementation is fail-closed: host guards raise errors rather than falling back to a generic Torch compute path, because a silently wrong value in an attention kernel is worse than a hard failure. Second, the naming split — the title says "qwen4" while the description names Qwen3.8-Flash-Next — is itself a signal. The vLLM side is treating the preview model as the first instance of the qwen4 line, and building kernels under the family name that will cover whatever comes after it.

For context on what a "fuse op" is: an inference engine normally executes a model's graph one kernel launch per operation, and every launch carries overhead — memory round-trips between kernels, dispatch cost. Fusing several element-wise and state operations into a single kernel cuts that overhead, which is exactly where a new architecture's cost and latency get won or lost. A model built on operators that have no fused implementation anywhere will be slow and memory-hungry no matter how clever the training; the first engine that fuses them well sets the model's real-world price.

Reading the kernel names as a blueprint

None of these operators exist in the standard transformer opset, which is why they need new kernels at all — and each one maps to a specific piece of the architecture Aliba​ba previewed in Qwen3.8-Flash-Next:

• HyperConnection → gated residual connections. The single residual stream is widened to four parallel branches with element-wise dynamic gating, letting the model decide how much to read from and write to each branch; residual state can be stored in FP8 to cut memory traffic.

• QSA → Qwe​n Sparse Attention. A lightweight indexer compresses keys into micro-blocks at a compression ratio of 4, scores them, and keeps the best 512 blocks — roughly 2,051 logical token positions — while the final softmax and value aggregation still run on the original, uncompressed K/V. Twelve of the 48 layers are QSA; the other 36 are GDN linear-attention layers with a fixed-size recurrent state and no growing KV cache.

• PLE state operations → n-gram memory. The 51-billion-parameter lookup table — "Per-Layer Embeddings" in the SGLang write-up — is hash-addressed from the current token plus the preceding two or three, fetching 16 rows per token. Its positions are precomputable, so it can live in host memory and be prefetched asynchronously instead of occupying GPU VRAM.

• The MoE body the kernels sit in: 125B total parameters with roughly 6B active per token — under 5% activation — routed over 512 experts with top-10 routing.

Two caveats keep this honest. First, the architecture above is read from Qwen3.8-Flash-Next's model card and community teardowns, not from any Qwe​n 4 document — it is evidence about the family, not a spec of the family. Second, none of it changes the core fact: Qwe​n 4 does not exist in any catalog.

A separate line of evidence is the vision path, worth keeping apart from the kernel reading above. Independent of any operator mapping, vLLM's own model registry on main lists a qwen4_exp entry — Qwen4ExpForConditionalGeneration — registered as a multimodal model with a Qwen3-VL vision tower and a Qwen3VLMultiModalProcessor. The Rust-frontend PR (#54813) that wires that config into the engine's multimodal crate, the code path that turns image inputs into tokens, merged on September 1, 2026 — a stronger statement than the same PR being open, because the multimodal path for the qwen4 line is now in the engine rather than in someone's fork. It is still not a release: the model that registry entry describes has no weights and no API. But it is the first engineering-level confirmation that the next Qwe​n generation takes image input, and the "multimodal" label on the architecture is no longer just a line on the preview's model card.

Why the architecture changes the cost picture

If Qwe​n 4 ships on this architecture, the economics are different from a standard dense — or even ordinary MoE — transformer, and that difference is the reason the kernel work is worth watching.

• Memory and compute split. Memory cost tracks total parameters — all 125B must be resident to serve the model. Compute cost tracks active parameters — only ~6B run per forward pass. That separation is what lets a "125B" model serve at the cost of a much smaller one.

• The KV cache stops growing on most layers. With 36 of 48 layers using fixed-size recurrent-state attention, the growing key/value cache is confined to 12 layers. At a 262K-token native context — 1M via YaRN — that is a large part of why long-context serving becomes affordable at all.

• The big table is offloadable. The n-gram table is the single largest memory consumer at 51B parameters, and it does not need to sit on the GPU. SGLang's day-0 work measured offloading the BF16 table to pinned host memory freeing about 23 GiB per GPU on H200 and increasing allocated KV capacity by roughly 78% — with bit-identical outputs. On the vLLM side the same offload landed in stages rather than in one PR: the original support PR (#53899) was closed without merging on September 21, while the capability itself arrived through separate merges — #54371, UVA-based PLE offload with Engram tensor parallelism, on September 9, and a ROCm path (#57497) on September 25.

None of these numbers is independently audited yet — the 7.6×/4.9× long-context speedup claims, the 540 tok/s decode figure from the SGLang post, the "1/9 of Qwen3.7-Plus" training-cost claim — and they should be read as vendor- or community-reported until a neutral index reproduces them.

The race to serve it

Both major engines have qwen4 work in flight — SGLang published day-0 support within days of the August 26 preview, and the work on each side has kept stacking up since, from config registration to kernel and memory engineering.

• SGLang published a day-0 post on August 26 in a collaboration across SGLang, Aliba​ba, NVIDIA, and AMD, shipping an NVFP4 checkpoint and claiming 540 tokens/s decode at TP4 on B200 with speculative decoding.

• On August 27 an sglang PR (#36585) added native Qwe​n4-Exp support to the engine itself — registering the Qwen4ExpForConditionalGeneration config and a text model built from the same GatedResidual, Gated DeltaNet, Qwe​n sparse attention, PLE, and MoE pieces. It is not merged: the PR comes from a non-maintainer contributor, the work credited to an autonomous agent named Argus, and it is still open and awaiting review. Its anchor is independently checkable, though — Qwen3.8-Flash-Next's own config declares model_type=qwen4_exp, and the reference implementation now sits in huggingface/transformers under models/qwen4_exp.

• On September 8 a draft sglang PR (#38492) opened against qwen4-main-squashed, an sglang branch dedicated to the Qwen4/Flash-Next line, and this one is not registration. It ports RecoverSSM — an accepted-state recovery path first built for Qwen3.8-Flash-Next on SM120 — for the architecture's multi-token-prediction (MTP) verification. MTP drafts roll through the Gated DeltaNet layers' fixed-size recurrent state, and instead of holding every intermediate speculative state in memory while a draft is verified, the engine reconstructs the accepted state from the committed state and the accepted token prefix. The mode is opt-in (a --gdn-mtp-cache-mode none flag; the current full-cache behavior stays the default), and it is early even by this story's standards: a draft last touched on September 14, whose current head still fails nine of its checks, with the author's own checklist open before it can be marked ready for review. The direction is the point — sglang is engineering the Qwen4 serving path, not just registering its config.

• vLLM's qwen4 line has moved from three open PRs to a mixed ledger in the weeks since this piece first ran. The Rust-frontend multimodal PR (#54813) merged on September 1. The PLE-offload work merged in two parts — #54371 on September 9 and #57497 on September 25 — while the original PLE-offload PR (#53899) was closed unmerged on September 21. The config has since gone upstream: #57387, which switches the qwen4_exp config and processor to the upstream Transformers implementations, merged on October 2. What is still open is the fuse-op PR that is the subject of this piece (#53909) — alongside a QSA ring-buffer bugfix filed on October 1 (#59528) and an SM86 (Ampere) pass over the 51-billion-parameter n-gram table open since September 26 (#58856), which is the kind of work someone does when a model is something people are trying to serve on older hardware rather than something nobody has run. The qwen4_exp config itself is registered in vLLM as multimodal: a Qwen3-VL vision tower paired with the Qwen4-Exp text model.

This is a race with a real prize. The engine that fuses the new operators best sets the served price for the Qwen4 generation, because kernel efficiency — not raw parameter count — is what a hybrid-attention model's token cost is made of.

What is actually out today: Qwen3.8-Flash-Next vs Qwen 4

It is worth being precise about the two things in this story, because the naming is doing a lot of work.

• Qwen3.8-Flash-Next is out. Open weights are on Hugging Face and ModelScope under the qwen-community-1.0 license, in standard and FP8 variants; the model card settles the spec sheet — 125B MoE, ~6B active, 51B n-gram table, 262K context, multimodal. It is the architecture preview, not the flagship.

• Qwen3.8-Flash is the served production variant Aliba​ba announced alongside it, listed on Qwe​n Cloud at ¥1 per million input tokens and ¥3 per million output tokens (roughly $0.16 / $0.47), with a default 1M-token context and built-in tools. Vendor-reported pricing, not yet in any independent index.

• Qwe​n 4 is not out. Aliba​ba named all four tiers on September 22, 2026 and said the family is in training and coming soon — the first official word on timing, and still not a schedule. No date, no card and no weights for any tier. The window doing the rounds now is October to early November, and it comes from watchlist threads and tracker pages rather than from Aliba​ba: the training status is company-stated, the timing is community-sourced. The September window that circulated in a July roadmap leak is spent, which is its own answer.

• The Qwe​n 4 27B tier is the one to watch for a non-benchmark reason: it is the only tier in the family with a track record of shipping open weights, and an announced 27B is the thing local-inference stacks plan around — which is why the missing size breakdown, licence and date matter more here than they do for the flagship.

The preview's benchmark table — DeepSWE 1.1 at 58.7, SWE-bench Pro at 62.5, LiveCodeBench v6 at 91.9 — is vendor-reported on the card and has not been independently reproduced. Plan around the architecture; treat the scores as a promise, not a measurement.

What a developer should do now

Nothing about your current stack changes today: Qwe​n 4 is not callable anywhere — Apsara named the tiers and put the family on a roadmap, and shipped none of it — and as of writing the Qwen3.8-Flash-Next open weights are still not on our own model list either — the closest things we pass through today are the served Qwen3.8 Flash variant at $0.15/$0.47 per million tokens and the Qwen3.8 27B open-weights route at $0.33/$2.40, which is the nearest thing to a locally-runnable Qwe​n in the catalogue while the successor generation is still a rumour. The roadmap line is the one thing worth reading twice, because it is the first time Aliba​ba has committed a successor generation to paper: if Qwe​n 4.5 and Qwe​n 5 really land in the 5-to-10-trillion-parameter band, the tier you can afford to self-host in that generation matters more, not less. If you want to feel the architecture before the family ships, the honest paths are to pull the weights yourself or wait for the preview to land behind an API. For anyone planning around the 27B tier, the same applies with one addition: it is worth tracking precisely because it is the tier most likely to be runnable without an API, and it is currently the least specified of the four.

When Qwen3.8-Flash-Next or a Qwe​n 4 model does show up in a provider catalog, the switching cost on OrcaRouter is roughly zero: one API for 200+ models, the provider's list price passed through at 0% markup, and a vendor price cut live on our side the same day. The same endpoint and key you already use for Qwen3.8 Max — callable today as qwen/qwen3.8-max at $2.00/$6.00 per million tokens, list price passed through — is the same endpoint a Qwe​n 4 call will use.

For an unproven architecture, automatic failover is the right first step: route a slice of traffic at the new model with a failover to Qwen3.8 Max (or whichever model the workload actually needs), and let your own traffic decide whether the preview lives up to the card — a routing-DSL edit, not a rewrite.

What we're watching

• Whether the fuse-op kernels merge (#53909), now the last of vLLM's original three qwen4 PRs still open — the multimodal front-end and the PLE-offload work have already landed, and the config has gone upstream. On the sglang side the native-support PR and the RecoverSSM draft are both still open. A merged kernel set is the strongest signal yet that the qwen4 line is on both runtimes' near-term roadmaps.

• Whether qwen4-main-squashed hardens into sglang's Qwen4 mainline — an integration branch that kernel and memory work keeps landing on is a stronger commitment than a single contributor PR, and it is worth watching what happens to it once the flagship has a date.

• Whether the vision path survives to release — the qwen4_exp multimodal registry entry and the Rust-frontend integration are the first hard signal that the flagship ships image input, and both are now in vLLM's mainline code rather than in a fork. The open question is whether the shipped model keeps that vision tower.

• The first independent numbers on Qwen3.8-Flash-Next, which will be the first test of whether the architecture's cost claims hold outside the vendor.

• A date or card for Qwe​n 4 itself. The family now has names, a tier list, a training status and a place on a published roadmap, and none of those is a schedule. The window the community is circulating is October to early November; the roadmap Aliba​ba published at Apsara carries no dates at all, and no Qwe​n 4 repository has appeared on Hugging Face. The next real signal is the first one with a date attached to it — a model card, an API identifier or weights.

• The Qwe​n 4 27B tier in particular — whether it ships open weights on the Qwen3.8-27B pattern, under what licence, and how quickly the conversion-and-fine-tune ecosystem follows.

• The license for the family — whether Qwe​n 4 repeats the Qwen3.8 Max pattern of open weights under a use-restricted license, or opens up further.

• The served-price race on the preview, which will tell us what the hybrid architecture's tokens actually cost in production.

FAQ

When does Qwe​n 4 come out? There is still no date, and there are now two official statements about timing rather than one. Aliba​ba Group CEO Eddie Wu confirmed at Apsara on September 22, 2026 that Qwe​n 4 is in training and that a successor line — Qwe​n 4.5 and Qwe​n 5 — is projected at 5 to 10 trillion parameters; Aliba​ba's English release repeats both. Neither statement carries a schedule. Three timing claims have circulated and none of them is Aliba​ba's: the July roadmap leak floating a September 2026 window, which has passed without a launch; and the October-to-early-November window now doing the rounds in watchlist posts and tracker pages, which is a reading of the lab's shipping cadence rather than a statement from the lab. The August 26 Qwen3.8-Flash-Next release is explicitly a preview of the architecture, not the flagship.

Is Qwe​n 4 the same model as Qwen3.8-Flash-Next? No. Qwen3.8-Flash-Next is a released open-weight model whose card calls it "A Preview of the Qwen4 Architecture." Qwe​n 4 is the unreleased family Aliba​ba named on September 22 — four tiers, no published specifications — and the architecture is shared while the product is not.

Can I use Qwe​n 4 today? No — nothing routes to a model that has no weights and no API, and the September 22 announcement did not change that. The nearest things you can actually touch are the Qwen3.8-Flash-Next open weights released August 26, or the Qwen3.8 Max API, live on OrcaRouter at $2.00/$6.00 per million tokens.

The kernel is the message — and the September 22 announcement, for all the stage time, did not overwrite it. vLLM does not spend review cycles building Triton kernels for operator families it expects to abandon, and Aliba​ba does not publish an architecture preview it expects to abandon either. Between the still-open "qwen4 fuse op" PR, the Qwen3.8-Flash-Next preview released on August 26, the qwen4_exp entry now sitting in vLLM's mainline registry with its config upstreamed and its multimodal integration merged, the PLE-offload work that landed on both CUDA and ROCm in September, and a draft sglang PR opened September 8 that is still tuning SM120 memory for the line's multi-token-prediction path, the shape of the next Qwe​n generation is no longer a rumor — it is written in the kernels and memory paths two runtimes are building for it. What Apsara added is a name for each tier, a training status, and a parameter band for the generation after it. What the community has added since is a window rather than a fact: October to early November, from the same watchlist threads. Aliba​ba has published neither. Plan for the architecture; don't bet on the dates.