Serving Qwen3.8-Flash-Next-Uncensored-FP8: a vLLM runbook for the block-FP8 build

Qwen3.8-Flash-Next-Uncensored-FP8 — the block-FP8 build of the abliterated Flash-Next — is the artifact you actually download when you serve this model on datacenter hardware, and it is the last one in the collection to get a runbook of its own. It lives at orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 on Hugging Face: the refusal direction removed from Qw​en's Qwen3.8-Flash-Next, then re-quantized offline to the exact FP8 scheme of the official Qwen3.8-Flash-Next-FP8 so vLLM serves it on the identical kernel path. It is the build anyone running this model on Hopper-class and newer GPUs will reach for, and the serving path has one flag that is easy to get wrong and hard to diagnose when it is.

First, the boundary, because readers keep blurring it and it changes everything below. Qwen3.8-Flash-Next-Uncensored and Qwen3.8-27B-Uncensored are two different models, not two builds of one model. Different base weights — Qwen3.8-Flash-Next against Qwen3.8-27B — different architectures, different weight drops, different Hugging Face collections. They share an abliteration technique and a family name; that is all. None of a 27B page's numbers transfer to this model, and if you arrived here from a 27B search, the 27B's own local runbook is a separate page with a separate set of decisions.

This page is the Flash-Next FP8 serving page and nothing else. The GGUF/MLX runbook covers the abliteration explanation for this model and the two consumer-hardware build lines; the technique behind the whole family is explained in the abliteration primer and the broader uncensored-LLM explainer; and Qwen3.8-27B-Uncensored-FP8, the sibling you might have been pointed at, has its own FP8 runbook. Here we stay on one question: how you serve the block-FP8 build, what breaks when you do it wrong, and what the card's own numbers do and do not tell you.

Before you start: the gate and the runtime

Two things gate this repo, and both produce failures that look like something else.

The first is access. The repository is gated: you must be logged in to Hugging Face and have accepted the repo's terms before any download works. The model page itself is readable without an account — the full card prose is public — but the weights are not. Plainly, without a logged-in session that has accepted the terms, hf download and vllm serve orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 both fail with an authentication error, not a friendly “you need to click Agree.” Do the one-time click-through first, then pull the ~186 GB with the hf CLI or let vLLM resolve the repo on first run.

The second is the runtime. The checkpoint registers under the qwen4_exp architecture (Qwen4ExpForConditionalGeneration), which stock vLLM and stock Transformers cannot load. You need the day-0 vLLM image and transformers 5.16+. This is the single most common “it doesn't load” failure across the community runbooks this week — not a corrupt download, a runtime that predates the architecture. The image is not optional; it is the path.

Hardware, so you can plan before you pull anything: the card's invocation targets an 8-GPU node, and the official vLLM recipe's guidance for the FP8 checkpoint applies here since the builds match tensor-for-tensor — on the order of 265 GB of GPU VRAM for a full-node deployment, with TP2 treated as the minimum on GB300-class and TEP4/TEP8 as the validated full-tray configurations.

Why the FP8 build exists — and why “identical kernel path” is the whole point

The abliterated BF16 weights are the source of truth; this repo is that model re-quantized offline, deliberately reproducing the official Qwen3.8-Flash-Next-FP8 recipe. The quantizer touches only the 512 routed-expert projections — experts.{e}.down/gate/up_proj — un-fusing them from the BF16 build's 3D layout and storing each as float8_e4m3fn weights plus BF16 weight_scale_inv scales in 128×128 blocks. Activations are per-token dynamic FP8; there is no calibration set. Everything else stays BF16: attention and linear_attn, the shared expert, the MoE router (mlp.gate), the Hyper-Connection mixers, embeddings, lm_head, the MTP speculative-decoding head, and the whole vision tower.

The “identical kernel path” line is more than marketing, and it deserves one sentence. The build was verified against the official FP8 checkpoint: block scales reproduce exactly (scale_relerr = 0) and the FP8 codes match to sub-ULP rounding. That is why vLLM runs it with the same block-scaled FP8 kernels and the same MTP speculative decoding as the official release — the tensors are effectively the same tensors, minus the refusal direction.

Concretely, that gives you ~186 GB across 131 shards (152,089 tensors, 75,264 of them FP8), 262,144 tokens of native context, the vision + video tower preserved byte-for-byte (333 visual.* tensors), and the MTP head intact. The weights were abliterated first — a single refusal direction estimated at layer 24 and orthogonalized out of 149 residual-writing tensors in float32, following Arditi et al. (2024) — and the MTP head's residual writers were edited consistently, so speculative decoding keeps working. That last detail is not obvious, and it is the difference between a head that accelerates decode and one that silently degrades it.

The one flag that makes or breaks the load

Serve this build without --enable-expert-parallel and you get a failure that looks like a shape bug, not a config mistake. It is the most reported serving failure for this checkpoint, and it is entirely deterministic.

Here is the arithmetic. The routed experts' fused gate+up projection has an intermediate size of 640. Block-FP8 quantizes in 128-wide blocks. Under plain tensor parallelism that 640 is split across ranks — 640 ÷ TP — and for the common TP degrees (2, 4, 8) the per-rank slice is not divisible by 128: TP8 gives 80, TP4 gives 160, TP2 gives 320. vLLM then refuses to load the weights with an error that reads like a shape mismatch: The output_size of gate's and up's weight = 80 is not divisible by weight quantization block_n = 128.

Expert parallelism fixes it by sharding the expert weights across expert-parallel ranks instead of tensor-parallel ranks, which preserves the FP8 block boundaries. That is why the flag is mandatory for this build: with --enable-expert-parallel, TP8 becomes a working TEP8. (It is harmless for the BF16 build, where there are no FP8 blocks to preserve.) The official vLLM recipe is explicit that plain TP8 is incompatible with the checkpoint's 128-wide quantization blocks, and a vLLM issue filed two days after the weights landed documents the identical failure on an 8×L40s node at TP2, TP4 and TP8. If a load dies with a shape-looking error, check the flag before you check the download.

The exact command

Here is the card's docker invocation, reproduced faithfully:

docker run -d --name flashnext --gpus all --ipc host -p 8000:8000 -v /path/to/Qwen3.8-Flash-Next-Uncensored-FP8:/model vllm/vllm-openai:qwen38-flash-next-x86_64-cu130 --model /model --served-model-name Qwen3.8-Flash-Next-Uncensored --tensor-parallel-size 8 --trust-remote-code --max-model-len 262144 --enable-expert-parallel --enable-auto-tool-choice --tool-call-parser qwen3_coder

Work through the flags that are not obvious:

• vllm/vllm-openai:qwen38-flash-next-x86_64-cu130 — the day-0 qwen4_exp image. This is not a generic vLLM; it is the architecture-specific image, and stock images predating qwen4_exp will not load the checkpoint at all.

• --trust-remote-code — loads the qwen4_exp modeling code shipped with the repo. Without it the loader refuses on principle.

• --max-model-len 262144 — matches the native context window. You want it explicit here rather than left to a default.

• --enable-expert-parallel — required for the FP8 build, for the reasons in the section above. The card notes it is harmless for BF16.

• --enable-auto-tool-choice --tool-call-parser qwen3_coder — switches on tool and function calling using the Qwen3-Coder XML format. Leave them off and the model still chats, but agentic tool use is off.

• --tensor-parallel-size 8 — the card's invocation assumes an 8-GPU node (8× Hopper-class). With --enable-expert-parallel that is a TEP8 deployment.

Once the container is up, the endpoint is OpenAI-compatible at :8000/v1. Set --served-model-name to whatever your clients expect; the card uses Qwen3.8-Flash-Next-Uncensored.

Alternatives, all in the card or corroborated by practitioners this week: vllm serve orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 directly once your HF session is authenticated; SGLang via the lmsysorg/sglang:qwen38flashnext image with --tp 8 --ep 8 — same expert-parallel requirement, same reason; and Transformers with pipeline("image-text-to-text", ...) on transformers 5.16+ if you want to script against the model rather than serve it.

What actually works when you serve it

The patterns in this section are community findings from practitioner runbooks and forum threads this week, not vendor guidance. Where more than one setup reports the same behaviour, it is worth treating as real:

• MTP speculative decoding works. Add --speculative-config '{"method":"mtp","num_speculative_tokens":3}' and vLLM uses the preserved MTP head. Multiple runbooks report MTP as the reason this model's decode stays usable despite its size.

• OOM on load? Offload the n-gram table. The 51B-parameter PLE n-gram embedding is the memory surprise in this architecture. VLLM_PLE_CPU_OFFLOAD=1 moves it to host RAM — give it at least ~51 GB there. The official recipe and community multi-node runbooks both reach for this flag.

• Vision is real, not vestigial. The vision + video tower is preserved byte-for-byte, so this stays a full vision-language model. Pass an image_url content part in a chat completion and the same endpoint serves image understanding; community OCR probes on this build report clean passes.

• Reasoning is on by default — and it changes the safety picture. The chat template enables thinking unless you say otherwise. Toggle per request with chat_template_kwargs={"enable_thinking": true|false}, and add a reasoning parser if you want thinking text separated from the answer. Because of this default, you are almost always serving the thinking-on model unless you explicitly turn it off.

• 262K native, 1M with a rope override. Native context is 262,144 tokens. Pushing toward 1M needs an explicit YaRN rope-scaling override plus an env var that lifts vLLM's max-model-len cap — and you should regression-test shorter-context quality first, because blind 4× extension is where long-context quality usually degrades.

What the card's numbers say — and what they don't

These are the vendor's own measurements on its own edit, published in the model card and measured on these exact weights served with vLLM against the official base under identical scripts and settings. Report them as what they are: indicative, and not an independent audit.

The headline is the refusal collapse with thinking off. On the card's harmful-prompt suite (n from 50 to 150 per benchmark), base refusal runs 64–100% and this build 0–2.7%: AdvBench 100%→2.0%, JailbreakBench 94%→0.0%, StrongREJECT 99.3%→1.3%, HarmBench 100%→1.3%, MaliciousInstruct 98%→0.0%, SimpleSafetyTests 64%→2.0%, ForbiddenQuestions 75.3%→2.7%, and a custom Chinese/English probe 63.6%→0.0%.

Now the honest half. The base model's own refusal rate collapses when thinking is enabled — AdvBench drops from 100% on the base to 7.0% with reasoning on — so the thinking-on comparison is much less dramatic: this build sits at 0.0% across the same suite, but it is shaving a small number off a number the base already reduced. Quote only the thinking-off figures and you are presenting the flattering half of the story, and that is precisely the half a safety evaluation must not rely on.

Over-refusal on benign prompts (XSTest-safe, n=250) drops from 9.6% on the base to 1.2% on this build with thinking off — a real improvement, since a model that refuses benign prompts is the quieter failure mode. Capability retention on MMLU / MMLU-Pro / GSM8K / CMMLU shows deltas of −2.0, −1.2, −1.3 and −0.6 points respectively, consistent with the claim that orthogonalizing one direction costs near-zero general capability. Tool calling, vision/OCR and reasoning are all reported as working on this build.

Two caveats sit over all of the above. The refusal metric comes from a rule-based opening-phrase classifier, which the card itself calls indicative rather than an LLM-judge or publication-grade number — a human panel or a judge model will not reproduce these exact figures. And the caveat column matters: on the thinking-off suite, roughly half to three-quarters of this build's outputs still open with a short disclaimer before complying. The model rarely refuses; it hedges. “Uncensored” here means it answers, not that it answers without preamble.

The safety section is not a formality

Read this before you pull the weights, not after.

This model has had its safety alignment substantially removed, and the mechanism is specific: a single refusal direction was estimated in the residual stream and orthogonalized out of every residual-writing matrix — 149 of them — computed in float32. The consequence is advertised, not incidental. The card is blunt that the model will comply with harmful, unethical, offensive, or illegal requests that the base Qwen3.8-Flash-Next would refuse, and that it has no meaningful built-in guardrails. It is released strictly for legitimate research — interpretability, AI-safety and refusal-mechanism study, red-teaming, robustness evaluation, and controlled experiments — and the user assumes full responsibility and liability for what it generates. Apache 2.0 governs what you may do with the weights.

Two things to get exactly right, because this build makes them easy to get wrong.

First, a jailbreak probe that “succeeds” against this model is not a passing safety evaluation. It is the advertised behaviour. If your evaluation's claim is “this model's safety was bypassed,” you have measured the design, not a vulnerability. What would actually be a finding is a refusal that survives the abliteration, or a capability regression — and the card's numbers suggest both are rare.

Second, the preserved attack surface is wider than text. The vision tower is intact byte-for-byte and tool calling works, so image input and agentic use are both live. A red-team plan that only probes text prompts misses the modalities this model actually exposes. And the refusal numbers above are a rule-based classifier on the vendor's own edit — they are not an independent audit of anything, safety included.

Do not deploy this to end users or into production without adding your own safety, moderation, and abuse-prevention layers. The repo's terms say it plainly, and it is not boilerplate: outputs do not reflect the views of the uploaders or of Qw​en / Ali​baba.

How to get a censored baseline for comparison

If your work is refusal-mechanism research or red-teaming, you almost certainly want the censored counterpart of this model side by side — the same architecture without the edit — to measure the delta. This uncensored build is local-only by design: the repo is gated and has no hosted inference deployment, which is intentional so that sensitive payloads never transit a third-party API.

For the hosted baseline, OrcaRouter routes the Qw​en line at provider list price with zero markup — Qwen3.8-Flash at $0.15 per million input tokens and $0.47 per million output, passed through as-is, with automatic failover and one key for 200+ models. A vendor price change shows up the same day. If you are weighing whether to run this build at all, or how much of your stack it can carry, that is the cheap way to hold the censored version up against it without a second contract or a second codebase.

Start here

Decision summary. You need: a Hugging Face account with the repo's terms accepted; a Hopper-class or newer node — the card's command targets 8 GPUs, and on the order of 265 GB of GPU VRAM per the official recipe's guidance for the matching FP8 checkpoint; the day-0 vLLM image and transformers 5.16+; and roughly 186 GB of disk for the weights.

Run order: accept the repo terms → download the weights → pull the day-0 image → serve with --enable-expert-parallel → verify with a request against :8000/v1/chat/completions → then start your evals. If a load fails with a shape-looking error, check the flag before you check the download.

And keep the frame. This is a research instrument, released on that condition. Its numbers are the vendor's own indicative measurements on its own edit. Its safety behavior is the point of the exercise, not a bug to be worked around. Serve it, measure it, and put your own moderation between it and anything human.

All five Flash-Next builds — BF16, GGUF, MLX, FP8 and NVFP4 — are collected in the Qwen3.8-Flash-Next-Uncensored collection on Hugging Face.

A different model, not another build of this one: Qwen3.8-27B-Uncensored is abliterated from a different base and has its own collection and its own runbooks.

These weights are local-only by design. For a hosted baseline to measure the abliterated build against, Qwen3.8-Flash is served on OrcaRouter at provider list price with 0% markup — the stock model, safety alignment intact.