ZoomGain-9B vs Gemma 4 12B: The Row That Is Missing From the Benchmark

Twenty-eight models were scored on the ZoomGain benchmark at release. Four closed-source frontier systems, four agentic vision models, four fine-grained specialists and sixteen open generalists. G​emma is not among them — not G​emma 4 12B, not Gemma 4 31B, not any of the E-series. That omission is the single most useful fact about comparing ZoomGain-9B and G​emma 4 12B, because it means every "ZoomGain vs G​emma" table you could build has a hole in it, and this piece is about what the two models actually are rather than a number that does not exist.

The two are close enough in size to invite the comparison and far enough apart in purpose to make it misleading. ZoomGain-9B is a 9B vision-language checkpoint from inclusionAI, post-trained by reinforcement learning on 9,000 crop-mined questions, released on 10 November 2026 with a paper but no announcement. G​emma 4 12B is G​oogle DeepMind's 11.95B encoder-free unified multimodal model, published on 23 May 2026, Apache 2.0, with 1.8 million downloads and 1,730 likes in the last month. One is a research instrument with one demonstrated capability. The other is a product with a licence, a documentation set, six quantisations and half a year of community use.

What each one actually is

G​emma 4 12B is the interesting architectural member of the G​emma 4 family. The E2B, E4B and 31B models use dedicated encoders — a roughly 150M-parameter vision tower on the small ones, roughly 550M on the 31B — and pass embeddings to the language model. The 12B does not. It projects raw image patches and audio waveforms directly into the decoder's embedding space through lightweight linear layers, which is what "Unified" means in its name, and it is the reason audio support exists on this model and not on the 31B.

Its shape: 48 layers, hidden size 3,840, intermediate 15,360, 16 query heads against 8 KV heads at head dimension 256, a 262,144-token vocabulary and 262,144 maximum positions. Forty of its forty-eight layers are sliding-window attention at 1,024 tokens; eight are full attention, and the final layer is always global. Three input modalities, text output, 256K context, and a native system role.

ZoomGain-9B is built the other way round. Its text trunk is Qwen3.5-9B — 32 layers, hidden 4,096, intermediate 12,288, a 3:1 schedule of linear to full attention, 262,144 maximum positions — with a vision tower bolted on by the base model rather than invented here. On top of that trunk the authors ran GRPO on 9,000 questions whose answers depend on details visible only when you zoom into a region. The model was not trained as a general assistant and is not presented as one.

• Parameters — G​emma 4 12B 11.95B vs ZoomGain-9B roughly 9.4B (18.82 GB in BF16 against the 12B's file size).

• Modalities — G​emma 4 12B takes text, image, video and audio; ZoomGain-9B takes text and image. Audio is not a gap in ZoomGain's implementation, it is absent from its config.

• Context — G​emma 4 12B documents 256K. ZoomGain-9B's card documents nothing; its config inherits 262,144 positions from the Qwen3.5-9B trunk, which is not the same as a tested property of the fine-tuned checkpoint.

• Licence — G​emma 4 12B Apache 2.0 with a named licence file. ZoomGain-9B: code Apache 2.0, checkpoint terms deferred to a card that carries no licence tag.

• Distribution — G​emma 4 12B ships as BF16, QAT q4_0 GGUF at 6.98 GB with a 0.18 GB projector, and builds from ggml-org, unsloth and litert; ZoomGain-9B ships as BF16 plus GGUF at F16, Q8_0 and Q4_K_M from 11 November.

• Documentation — G​emma 4 12B has a launch blog, a technical report, framework cookbooks and a licence page. ZoomGain-9B has a paper, a repository and a card that does not state a context length.

Why the missing row is not an oversight you can wave away

The ZoomGain benchmark is built to expose one failure: a model that cannot resolve a small detail inside a large frame. The authors score attribute recognition, counting and OCR over 1,000 questions and 941 images, in English and Chinese, with exact match on multiple choice and on integers, and ANLS at a 0.7 floor for OCR. On that test the best of twenty-eight models reaches 61.08, and only five clear 50.

G​emma 4 12B has not been run on it. Neither has any other G​emma. That leaves you with two options, and only one of them is honest. You can line up G​emma's MMMU Pro 69.1 and MATH-Vision 79.7 against ZoomGain's held-out average of 66.10 and call it a comparison, which is comparing a broad multimodal suite against an average of thirteen perception benchmarks and produces a number with no meaning. Or you can say what is true: nobody has measured these two on the same ruler, and if the question matters to you, the benchmark is downloadable and you can run it yourself in an afternoon.

The second option is cheaper than it sounds. ZoomGainBench ships as annotations, images and an evaluator; the evaluation takes an OpenAI-compatible endpoint or a local checkpoint, runs at temperature 0 with thinking disabled, and reports per-question results. G​emma 4 12B runs under vLLM or llama.cpp. That is a real head-to-head you could produce, and the fact that the benchmark's authors did not include a G​emma is itself informative — the model was available for six months before this benchmark was published.

Where G​emma 4 12B has the numbers, and what they mean

These are G​oogle's own figures from the model card, unreproduced by a neutral lab:

• MMMU Pro 69.1 — the multimodal reasoning suite, and the closest thing on G​emma's card to a visual competence claim.

• MATH-Vision 79.7 — mathematical reasoning over diagrams and images.

• OmniDocBench 1.5 average edit distance 0.164 — document parsing, where lower is better; the 31B reaches 0.131.

• MMLU Pro 77.2, GPQA Diamond 78.8, AIME 2026 77.5 without tools — the text side, which ZoomGain-9B does not claim at all.

• τ² average 69.0 and native function calling — agentic tool use.

• MRCR v2 eight-needle at 128K, 43.4% — long-context retrieval, and the number that most undercuts the 256K headline.

The last one deserves emphasis. A 256K window is not 256K of usable attention, and G​oogle publishing a retrieval score that drops to 43.4% at 128K is more candour than most cards offer. If your workload is "find one fact in a large document", G​emma 4 12B's window is real but its recall at depth is mediocre, and no equivalent measurement exists for ZoomGain-9B because no long-context evaluation was run on it.

Where ZoomGain-9B has the numbers, and the caveat attached to each

All author-reported, on the released harness:

• HR-Bench 4K 90.25, up from 82.13 for the Qwen3.5-9B it came from. High-resolution perception, and the headline claim of the release.

• V*Bench 93.19, up from 86.91. Small-object detection in cluttered scenes.

• MMVP 70.00, up from 60.00. The paired-image test where two nearly identical pictures have different correct answers — the closest existing analogue to what ZoomGain was trained for.

• ZoomGainBench 55.48, up from 32.13. The held-in column, inflated by construction, and the authors say so.

• Held-out average 66.10 across thirteen benchmarks, up from 59.75. The number that means something.

The caveats, in order of how much they should worry you. First, nobody outside the author list has reproduced any of it. Second, the benchmark and the training data come from one pipeline, which is why the paper excludes its own benchmark from the held-out average — a discipline worth crediting. Third, the training set is 9,000 records, small enough that the gains may be narrow even where they are real. Fourth, the model has no hosted endpoint and, at the time of writing, an empty provider mapping on the Hub.

Cost, in the terms that actually decide it

G​emma 4 12B at QAT q4_0 is 6.98 GB plus a 0.18 GB projector. That fits a 16 GB laptop with room to spare and an 8 GB card with effort. ZoomGain-9B at Q4_K_M is 5.63 GB plus 0.52 GB of projector; at Q8_0 it is 9.53 GB plus 0.62 GB. The 9B is the smaller download at every precision because it is the smaller model, and its linear-attention-heavy trunk means a shorter KV cache at long sequence lengths, which matters if you are feeding it high-resolution images rather than long text.

Neither can be called through us today. G​emma 4 12B is not in the OrcaRouter catalogue, and neither is ZoomGain-9B, so the honest statement is that both are self-host or vendor-API decisions. What we do host is Gemma 4 26B A4B at $0.06 per million input tokens and $0.33 per million output with a 262K window — the same family, four times the active parameter budget, and a legitimate way to find out whether G​emma's document-parsing profile suits your pipeline before you download anything. One key covers that and 200-plus other models at list price with no markup, so a bake-off across a vision endpoint and a cheap text model costs one afternoon rather than three contracts.

Choose by job, not by rank

If the task is a small detail inside a big image — a serial number on a device, the text on a distant sign, the colour of one object among forty, a count of windows on a facade — ZoomGain-9B is the one built for it, with the caveat that its evidence is self-reported and its licence needs reading. Its architecture is optimised for exactly the failure mode you are hitting.

If the task is anything else visual — reading a multi-page document, describing a scene, answering a question about a chart, taking audio as well as images, calling a function afterwards — G​emma 4 12B is the finished product, and the comparison stops being close. It has 140-language pretraining, native tool calling, a 256K window with a published retrieval curve, six quantisations and a licence you can hand to legal without an explanation.

If the task is both, the honest answer is that these two are layers of one pipeline rather than alternatives. The realistic 2026 document agent parses with a general multimodal model and escalates the crops that matter to a perception specialist — and the escalation is where the interesting engineering sits, because it means deciding when a detail is worth a second call. That is a routing problem, and it is why we care about it: a rule that sends the cheap call to the cheap model and the escalation to whatever endpoint clears your bar is one config change, and failover means a stalled specialist does not stall the pipeline.

The verdict, stated in the form the evidence supports

G​emma 4 12B is the safer purchase by every measure that is not a perception benchmark: licence, documentation, distribution, breadth, community, and the fact that G​oogle publishes numbers you can argue with. ZoomGain-9B is the more interesting model, and plausibly the better tool for one specific job, on evidence that has not yet been checked by anyone who did not write the paper.

The right next step depends on which of those sentences you found yourself agreeing with. If it was the first, download the QAT GGUF and stop reading. If it was the second, download ZoomGainBench along with the weights — 941 images, 1,000 questions, an evaluator and a number you can produce yourself — and run G​emma through it. You would then be the first person to have done so, and you would know something about both models that neither vendor currently does.