MiniCPM-V 4.7 Puts a 35B Vision Model in 3 GB: Untangling the 100 tok/s Claim

A 35-billion-parameter model cannot fit into three gigabytes. MiniCPM-V 4.7 35B-A3B runs in roughly three gigabytes anyway, and both statements are true — which is the whole trick, and the reason the numbers circulating about this release are worth slowing down for. The model is OpenBMB's first sparse mixture-of-experts entry in the MiniCPM-V line: 35.2 billion parameters stored, about 3 billion active per token, Apache-2.0, weights on Hugging Face. It reads images, multi-image sets and video, and answers in text. Since 10 October it also runs in upstream llama.cpp, which is where most people met it. What follows separates the three numbers everyone is repeating — 3 GB, 35B, and the 100 tok/s figure — and shows which of them OpenBMB actually published. 200-plus hosted models, one key, provider list price passed through at 0% markup. See what OrcaRouter routes The open-weight checkpoint stays on your hardware; the hosted escape hatch is one endpoint. The documented AI failures and agent incidents behind releases like this one are collected in the OrcaRouter AI Incident Archive where each entry carries its own sources, severity, confidence grade and disputed flag.

Three gigabytes, thirty-five billion parameters, and no contradiction

The arithmetic only looks impossible if you assume "in memory" means "all of it". MiniCPM-V 4.7 35B-A3B keeps about 3 billion parameters resident and streams the rest of the experts in as the router asks for them — the model card's own framing is Lookahead Expert Scheduling, a cross-layer routing predictor that prefetches the experts the next layer is likely to need, overlapping that load with computation already in flight. The card is careful to say prediction only drives prefetch and scheduling; tokens still follow the model's real routing, and an expert the predictor missed is loaded and evaluated like any other.

That distinction has a measurable consequence. The GGUF repository publishes the whole model in every common quantisation, and none of those files is three gigabytes:

• Q4_K_M — 21.2 GB on disk.

• Q8_0 — 36.9 GB.

• F16 — 69.4 GB.

• Vision projector — 0.9 GB at 4x compression, 1.1 GB at f16.

So "3 GB" is a resident-memory figure under expert streaming on a phone, not a download size and not a VRAM requirement for a desktop. If you are planning a deployment, the number that governs your disk and your first load is 21.2 GB at Q4_K_M, and the number that governs your steady-state footprint on a phone is the vendor's ~3 GB. Both are in the release. They answer different questions, and most of the coverage has collapsed them into one.

Where 100 tok/s comes from, and where it does not

Here is the part worth being blunt about. OpenBMB does not publish a 100 tok/s figure for MiniCPM-V 4.7. The number on the model card is 13+ tokens/s, on an iPhone 17 Pro Max, at roughly 3 GB of memory. That is the only on-device throughput claim the release makes, and it is vendor-measured on hardware nobody outside the lab has reproduced.

Higher numbers do exist, and they come from a different artefact. OpenBMB's llama.cpp-omni fork — the repository that carries the on-device optimisations — publishes its own table for the 35B-A3B target, stacked by which optimisations are switched on:

• iPhone 18 Pro Max, 12 GB — 3.38 tok/s baseline, 11.34 with Lookahead Expert Scheduling, 17.16 with mixed precision added, 20.51 with the DSpark draft head as well.

• iPad Pro 11-inch M4, 16 GB — 2.97 tok/s baseline, 9.81 with LES, 14.92 with mixed precision, 17.39 with DSpark.

Those are the highest published on-device figures for this model, and the top of the ladder is 20.51 tok/s — a fifth of 100. The gap matters because it changes what the model is for. At 13 tok/s a phone app answers a question about a photo in a couple of seconds. At 100 tok/s you would be considering it as a streaming first-pass reader for video. The first is a real product. The second is not what shipped, and anyone budgeting against the tweet's number will be disappointed by a factor of five.

The honest reading of the 100 tok/s claim is that it is a compression of three separate facts — 3 GB, runs locally, faster than the last generation by a wide margin — into one number that none of the sources contains. The generation-over-generation improvement is real and large: 3.38 to 20.51 tok/s is a 6x move on the same class of device, achieved by stacking three techniques rather than by a bigger chip.

The release was not one day, and the dates matter

MiniCPM-V 4.7 surfaced the way a lot of open-weight releases do now: files first, explanation later, and the framework support somewhere in between.

• 6 October, 17:40 UTC — the 1B model and its AWQ, GPTQ and int4 variants appear on Hugging Face, plus the GGUF builds at 18:05.

• 7 October, 07:07 UTC — the 35B-A3B repository lands. Still no model card.

• 8 October — DSpark speculative-decoding heads for both sizes, and an FP8 build of the 35B.

• 10 October, 18:42 UTC — llama.cpp merges model support.

• 12 October — OpenBMB commits the model card, declares Apache-2.0, and announces the release as a documented product.

• 15 October — the DSpark heads are linked from the main card, with GGUF drafts for both targets.

That order explains a lot of confusion in the secondary coverage. Someone who met the model through llama.cpp in the second week of October was running a release that officially did not exist yet, with no licence tag and no benchmark chart, and their throughput numbers reflect the framework's own quantisation choices rather than anything OpenBMB benchmarked.

llama.cpp support: merged, and not quite finished

The upstream pull request that added MiniCPM-V 4.7 is unusually readable about what changed, and the answer is: almost nothing architecturally. The model is near-identical to MiniCPM-V 4.6; the delta is 3D RoPE, a three-value position per token — time, row, column — instead of the single scalar most vision-language models use. Support extends the existing 4.6 path to carry that extra dimension. The PR also noted the model as "upcoming" when it was opened on 25 September, which is a fair hint about how early the weights were in circulation.

The residue is worth knowing before you build a grounding feature on it. Since that merge, upstream llama.cpp assigns the image marker tokens the wrong positions in MiniCPM-V 4.7 prompts: the image tokens themselves land correctly, but <slice>, </slice> and the end-of-row markers are numbered like plain text rather than given their tile corner coordinates as the reference get_rope_index implementation does. The visible symptom is that returned coordinates sit systematically low when you ask where something is in a picture. A fix with a unit test exists as an open pull request, dated 14 October, and as of 20 October it is still open. If your workload is "describe this", you will not notice. If it is "give me the bounding box", you will, and you should either wait for the merge or use the fork that carries the corrected path.

One more thing survives the round trip: for the 1B model, the vision projector is the bigger download. Q4_K_M weights are 672 MB; the f16 projector is 1.11 GB. The smallest useful 1B deployment is therefore about 1.8 GB of files, most of it eyes rather than language — a detail that any "1B means 1 GB" expectation gets wrong.

What to run, and on what

If the goal is to try it this week, the lowest-friction path is llama.cpp or Ollama against the GGUF repositories, pointing the multimodal client at the matching mmproj file. On a 24 GB GPU, Q4_K_M at 21.2 GB leaves very little headroom for context — an 8K-token window plus image tiles will not fit comfortably, so plan on Q4_K_S or a smaller quantisation if you also want a long prompt. On a Mac with 32 GB or more, Q8_0 is viable and is the closest thing to the reference behaviour. On a phone, only the vendor's own apps and the llama.cpp-omni fork carry the expert-scheduling work; stock llama.cpp gets you correctness, not the 20 tok/s.

Where a routing layer earns its place here is on the far side of the local call. A 3B-active model on a phone is excellent at the routine frame — read this receipt, does this shelf look stocked, what does this sign say — and it will occasionally meet a page it cannot finish. OrcaRouter puts 200-plus hosted models behind one key, passes each provider's list price through at 0% markup so a vendor price change is live the same day, and fails over automatically when an upstream degrades. The pattern that works is local-first with a hosted escape hatch, not local-only. The open-weight checkpoint stays yours to host; the router handles everything the phone hands off.

MiniCPM-V 4.7 is not served by OrcaRouter, and we are not going to imply otherwise. It is a checkpoint you run.

What would settle the three numbers

An independent measurement of the 35B model on a named phone, with the app and the quantisation stated, would settle the throughput question — including whether 20.51 tok/s survives someone else's stopwatch. A published definition of the 3 GB figure — resident set, peak allocation, or something else — would settle the memory question. And a per-benchmark table with real scores, rather than the five linked charts the card currently ships, would let the "better than Qwen3.6 35B-A3B in less time" claim be checked against the weighting OpenBMB chose. None of those exists yet. Until they do, the accurate sentence is the boring one: a 35B sparse model that runs on a phone at a reported 13-plus tokens per second in about 3 GB, with a 21 GB download and a merged framework path that still has one open coordinate bug.