What Is DeepInfer? DeepSeek's Inference Engine, and the Parts It Did Not Ship

DeepInfer is the inference framework DeepSeek published on 16 October 2026 under Apache-2.0, and it serves exactly one model: DeepSeek-V4.1-Flash, with DeepSeek-V4.1-Flash-1017 as the default checkpoint. That single-model scope is the first thing to understand, because it explains almost everything else about the repository. This is not a general serving framework that happens to include a DeepSeek configuration. It is the code path DeepSeek runs in production, cut down to the parts that generalise, and it is honest about the cut: the README's own list of what a complete inference service needs, and what this release does not include, names a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas. What you get is the compute path — kernels, parallelism, speculative decoding, scheduling, structured output. What you do not get is the storage and routing layer that makes the compute path cheap to operate at scale. DeepSeek says why: those components are tightly coupled to internal infrastructure, so the design is described in a published write-up while the code stays in-house.

What is actually in the repository

The published feature list reads as a tour of the bottlenecks in serving a large mixture-of-experts model. There are the kernel libraries DeepSeek has released before — FlashMLA for attention, DeepGEMM for the matrix multiplies, DeepEP for expert-parallel communication, plus TileKernels and DeepSelect. There is the parallelism scheme: attention runs as data parallelism with balanced context parallelism, the MoE half runs as expert parallelism with an expert-parallel load balancer. There is DSpark, the speculative decoding stack, with verification lengths that are scheduled rather than fixed. There is SWA bounded replay, which is the mechanism that keeps a sliding-window attention cache from forcing a full recomputation on every cache miss. There is asynchronous batch scheduling, batch-invariant inference with deterministic sampling, and structured output constrained by regex, JSON schema or EBNF.

The entry point is deliberately small. After setting DEEPINFER_ACCELERATOR to nvidia or the Ascend equivalent and running build.sh, the repository ships examples/offline_inference.py, which runs the default checkpoint end to end. DEEPINFER_HF_MODEL_DIR points it at a different weights directory. The published benchmarks were measured on four SM100 GPUs for prefill and 64 SM100 GPUs for decode, which is the shape of the deployment the engine was written for and not a laptop-shaped demo.

The README also states where the code came from, which is worth knowing before reading it. The engine was originally built primarily on vLLM, with design inspiration taken from SGLang and TensorRT-LLM. That lineage shows in the structure — the scheduler, the paged attention interface and the sampling path will look familiar to anyone who has read those projects — and it explains why the release is a fork of a serving stack rather than an architecture written from scratch. What DeepSeek added is the part that is specific to its own model: the disaggregated deployment, the wide expert parallelism with its balancer, the sliding-window replay and DSpark.

What it does not contain, and why that matters more

The three missing pieces are not small omissions. Each is a system that has to exist before the engine serves production traffic, and each is expensive in a different way.

• The KV cache system. Prefix reuse is where most of the money is in a chat workload, because a cache hit replaces a prefill. Without a cache store, every request recomputes its prompt. DeepSeek's own write-up describes the store it uses: a dedicated 3FS cluster rather than local host memory, precisely so that any prefill machine can hit any prefix and cache capacity can be scaled independently of compute.

• VE/Prefill/Decode disaggregation. The README notes that most of the engine-side functionality already exists in the release; what is missing is the deployment. Disaggregating the vision encoder, prefill and decode lets you size each stage against its own bottleneck and control time-to-first-token separately from time-between-tokens. That is an infrastructure decision, not a flag.

• Load balancing across data-parallel replicas. DeepSeek runs four balancers — prefill, decode, vision encoder, and expert-parallel — and describes all four in the write-up. None of the routing logic ships. Without it, a replica fleet is only as good as whatever router you put in front of it, and a naive round-robin router will destroy the cache hit rate the store was built to protect.

Read together, the omission tells you what kind of artefact this is. DeepInfer is a reference implementation of the compute path, published so that the kernel and scheduling work can be inspected and reused, not a distribution you install and point traffic at.

The hardware gate is the real filter

DeepInfer targets NVIDIA SM100 and Huawei Ascend 950. There is no Ampere path and no Hopper path, which excludes most of the GPUs in most clusters today. The rest of the toolchain is equally specific: Linux only, GCC 13 or newer, Python 3.12, CUDA 13 for NVIDIA or CANN 9.2 for Ascend. And there are no releases at all — the repository had zero published releases when it was read on 21 October 2026, so there is nothing to download and nothing to pin. You clone it and build it against your own toolchain.

That combination is unusual for an open-source release with this much attention, and it is best read as an accurate statement of scope rather than an oversight. The kernels in the repository are written for a specific instruction set and a specific interconnect; the numbers in the README come from that hardware. Porting the compute path down a generation is exactly the kind of work that a community does quickly and an engineering organisation cannot promise. Within days of the release a third-party port targeting older NVIDIA parts appeared on GitHub, which is a useful signal about demand and no signal at all about whether the published throughput figures survive the port.

The deployment shape DeepSeek actually runs

The architecture write-up is where the repository's assumptions become legible. The system is disaggregated into a vision encoder, prefill and decode, plus dedicated storage nodes. Prefill uses expert parallelism of 2 to 8 with matching data parallelism, choosing the smallest configuration HBM allows. Decode goes the other way: expert parallelism and data parallelism of 64 to 128, which DeepSeek calls Wide-EP, using the largest configuration the available GPUs allow. Tensor parallelism and context parallelism are not used at all. The vision encoder runs one GPU per instance, with cheaper GPUs recommended, because the encoder is not where the money goes.

The smallest deployment that makes sense follows from those numbers. One prefill GPU and one decode GPU will run, and in practice the ratio between the two pools is roughly one to one. But since decode wants 64 GPUs or more to get the expert-parallel benefit, the smallest efficient unit is 64 prefill GPUs plus 64 decode GPUs. That is the number to hold on to when reading any claim about DeepInfer's efficiency: the figures describe a 128-GPU deployment, and half of it is one stage.

Splitting prefill and decode across data centres is technically possible because DeepSeek-V4.1-Flash compresses its KV cache substantially, and DeepSeek's own deployment does put inference nodes in different but adjacent data centres. The budget is roughly 400 Gbps of interconnect per eight GPUs, and the write-up is candid about the reliability cost — fibre between buildings is easy to cut.

Why decode is spread so wide

Wide expert parallelism is the single most consequential choice in the design, and it is easy to misread as a scaling trick. In a mixture-of-experts layer, every token touches a small number of experts, so the cost of a decode step is dominated by moving expert weights and activations rather than by arithmetic intensity. Spreading the experts across 64 or 128 GPUs reduces the per-step memory traffic each GPU contributes, which raises tokens per second if — and this is the condition — the load across experts stays even. That is what the expert-parallel load balancer is for.

DeepSeek's settings are specific enough to copy. For prefill, each GPU holds four redundant experts beyond the ones it owns, statistics are collected over a 100-step window, and experts are reassigned every 1,000 steps, with weights shuffled within the group over NVLink shared memory. Measured imbalance — slowest rank divided by the mean — came out at 1.074 for an EP4/DP4 configuration and 1.045 for EP2/DP2. For decode the settings stretch out: one redundant expert per GPU, a 1,000-step statistics window, reassignment every 10,000 steps, and a measured imbalance of 1.090. The difference between the two is the point. Prefill sees bursty, heterogeneous work and needs fast rebalancing; decode sees a steady stream and can afford a long window, so it rebalances rarely and pays almost nothing for the privilege.

The routing side has the same character. Prefill requests are bucketed by total context length into four tiers — under 16K, 16 to 64K, 64 to 256K, and 256K and above — and engines are assigned to tiers according to each tier's share of compute demand in the previous round. Within a GPU, the chunk size for each prefill step is not fixed: a fitted quadratic cost model predicts how long a step will take given the number of sequence tokens and the prefix already computed, and the scheduler solves for the largest chunk that fits a 32,768-token budget inside the remaining time. The decode balancer balances two dimensions at once, request count and total KV cache length, sending short requests to GPUs with low concurrency and long cache, and long requests to GPUs with small cache. DeepSeek's own summary of how much all of this buys: roughly 80 percent of a perfectly balanced offline benchmark.

The cache, and the three-day question

The KV cache lives on a dedicated 3FS cluster, not in local host memory, and that decision drives the rest of the storage design. Data is stored with 10+2 erasure coding, so each coding group survives two simultaneous node failures; a read issues requests to all 11 nodes in the group and any 10 responses are enough to decode. A garbage-collection worker runs on every storage node and evicts under LRU past a watermark. The recommended server has 20 solid-state drives of 30.73 TB each and sustains at least 40 GB/s of read and 5 GB/s of write, sized to retain roughly 24 hours of data per server. In practice the cluster runs at about 14.4 TB of logical cache per online GPU, with physical capacity larger than that.

Residency is where the trade-off becomes visible. There is no expiry logic on the cache side; the TTL is set on the storage side at three days for agent traffic and seven days for chatbot traffic, on a cluster holding roughly 100 petabytes of solid-state storage. The split exists because chatbot traffic has a much heavier long tail of hits — a small amount of cache is read many times over, so losing it costs more overall hit rate than the volume suggests. DeepSeek's measured cost of trimming chatbot retention to three days is 0.5 to 1 percent of cache hit rate. That is a small number that happens to represent a large amount of recomputed prefill.

Sliding-window attention gets a separate cache with different rules. The SWA cache does not enter the paged blocks; each sequence occupies one fixed-size slot in host memory, and for DeepSeek-V4.1-Flash the stored layers are the first 20. Only that region persists — 20 layers by 134 tokens by 528 bytes, which is 1,415,040 bytes per sequence in an MXFP8 format. Nothing goes to the storage cluster; it stays in the memory of the machine that produced it, with an in-memory directory recording which machine holds what, and a cross-machine hit pulls it back over a 400 Gbps VLAN.

The two mechanisms that produce the headline numbers

Bounded replay is the more interesting of the two because it takes an unbounded cost and makes it constant. Normally, a sliding-window cache miss means recomputing the window from the beginning, and the cost grows with context length and layer count. DeepSeek instead recomputes only the most recent 128 tokens and truncates the attention window to that segment, producing an approximate SWA cache at a fixed price. On the encoder side this decouples the prefix-cache hit length from the SWA cache entirely: if the SWA cache is missing, the last 128 tokens of the hit prefix are replayed while the block cache is reused untouched.

The decoder side is where the saving is large. In this architecture the decoder layers' block cache is projected from the last encoder layer's hidden state, so only the decoder SWA cache has to be computed per layer — and reconstructing it exactly would mean pushing the last 2,560 tokens through 20 decoder layers on every prefill. Replaying 128 tokens instead, and using the result for decode only rather than letting it enter the prefix cache, nearly halves total prefill computation. That single change is worth more than most kernel optimisations, and it is the kind of thing only a team that owns the model architecture can do.

DSpark's contribution is smaller and more conventional. Draft tokens have different acceptance probabilities depending on the request and the position, so verifying a fixed five drafts per step wastes compute on tokens that will be rejected. A confidence head predicts each draft token's acceptance rate, and the scheduler sorts the drafts by expected return and verifies only the prefix that maximises throughput. In production the draft model still generates five tokens per step; the number verified drops from 5.00 to between 3.2 and 4.4, a cut of about 30 percent, while accepted length barely moves — 3.29 against 3.20. Read off EP64/DP64 benchmark data, that works out to 4 to 7 percent more throughput at around 60 concurrent sequences per rank. At exactly 60, throughput per rank goes from 13,730 to 14,466 tokens per second. Below about 16 concurrent sequences the technique is a small net loss, which is worth knowing before adopting it for a low-traffic deployment.

What the 155 trillion figure means

The most quoted number attached to this release is a single cluster serving 155 trillion tokens per day at overall revenue of 38 US dollars per GPU-hour at valley pricing. Both halves are vendor-reported and neither is a benchmark you can reproduce, which does not make them uninteresting — a revenue figure per GPU-hour is a genuinely different kind of claim from a throughput number, because it folds utilisation, pricing and cache hit rate into one line. It is also the figure most likely to be quoted without its qualifiers, so the qualifiers deserve writing down: one cluster, valley pricing, revenue rather than cost, and aggregate across prefill, decode and vision encoder.

Two smaller disclosures are more directly useful. The FlashMLA decoding kernel in the public repository is missing one hand-tuned assembly optimisation that the internal version has, which makes it 5 to 10 percent slower in isolation; DeepSeek states end-to-end performance is at most 5 percent apart. And the internal draft model for speculative decoding is newer than the published one, with the performance difference described as small. Both statements set an expectation about how much of the published performance a third party can reach by copying the code, and the answer is: most of it, provided the hardware matches.

The repository, as it stands

The capture above was made on 21 October 2026 and shows the page reading 16.3k stars. The GitHub API, queried the same day, returned 16,590 stars, 1,649 forks and 176 open issues, with issues disabled and discussion routed to GitHub Discussions instead. The star count has kept climbing since, so read every figure here as a timestamp rather than as a property of the code. Traffic of that size in under a week says more about interest in DeepSeek's serving stack than about the code's readiness, and the zero-releases count says the rest.

If you are thinking of running it yourself

The honest summary is that DeepInfer is worth reading closely and hard to operate. Four questions will tell you whether it is a fit.

• Do you have SM100 or Ascend 950? Everything else in the toolchain follows from this one. Without the right accelerator the published numbers do not transfer, and the community ports are unvalidated at production scale.

• Can you run 64 decode GPUs in one communication domain? Wide expert parallelism is not an optimisation you can skip; it is the configuration the decode path was designed around. Below that scale you are running a different system with the same name.

• Do you already own a prefix cache? If not, you are comparing DeepInfer's compute path against a competing stack that ships with one, and the comparison will look worse than the design deserves.

• Can you afford to write the routing layer? Prefill bucketing, decode balancing and expert reassignment are described well enough to reimplement, and reimplementing them is weeks of work that the repository does not do for you.

If you would rather not own any of the above

For most teams the practical question is not how to deploy DeepSeek-V4.1-Flash but where to call it. The model is available through a compatible API without building a 128-GPU cluster or reimplementing a load balancer, and that is the same model whose architecture this article has been describing — same context handling, same structured-output behaviour, without the disaggregation work.

The comparison to make is not which engine is faster in the abstract, because the published figures are vendor-reported measurements on hardware most readers do not have. It is whether the workload is one where a 3-day cache TTL and a 64-GPU decode pool change the price, or one where a request every few seconds through an API is simply cheaper than any of the operations described above. For the second case — which is most cases — the right amount of DeepInfer to use is none, and the right amount to read is all of it. Start from the DeepSeek V4.1 Flash model page if you want the endpoint; start from the repository's README if you want to know what the endpoint is doing.