A generated title card reading 'Aion-1.0-Instruct-2.3B vs Gemma 4 12B' with the subtitle 'Five times the weights and ten times the evidence', above a horizontal bar comparison of 2.29B against 11.95B parameters and three badges: '1 day of community evidence', '7 months of community evidence' and 'Both vendor-reported'.
Guides & Insights

Aion-1.0-Instruct-2.3B vs Gemma 4 12B: Five Times the Weights and Ten Times the Evidence

Author

jinhao.song

Date Published

Latest models · 20View all models →
Benchmarks: Artificial Analysis · updated daily
Back to all posts

Put Aion-1.0-Instruct-2.3B next to G​emma 4 12B and the parameter gap is the least interesting thing about them. Microsoft's checkpoint is 2.29 billion parameters and went up on Hugging Face on 18 November 2026 with no announcement; G​oogle DeepMind's is 11.95 billion, Apache 2.0, and has been downloadable since May 2026 with a model card that prints a full evaluation table and a community that has already pulled the weights more than a million times. The interesting gap is that one of these models has been examined by strangers and one has not been examined by anybody.

This is a comparison of two open releases rather than two APIs, so every claim below is checkable by downloading something. The Aion-1.0-Instruct-2.3B figures come from the model card and the config.json in its repository, both read directly today. The G​emma 4 12B figures come from G​oogle's model card, and its benchmark numbers are the vendor's own — reported, not independently reproduced, which is a distinction that matters for both models here.

The comparison table nobody can fill in evenly

Six dimensions, both sides, and a note where one side simply has nothing to put in the cell.

• Parameters — Aion-1.0-Instruct-2.3B is ~2.29B dense with tied embeddings; G​emma 4 12B is 11.95B dense across 48 layers.

• Licence — MIT for Aion, Apache 2.0 for G​emma. Commercially, both are effectively unrestricted; MIT is marginally simpler to comply with because it carries no patent grant to reason about.

• Context — 32,768 tokens against 256K, an eightfold difference that decides whether a whole repository fits in one prompt.

• Inputs — text only versus text, image and audio, with G​emma 4 12B's defining trick being that it has no separate encoders: image patches and audio waveforms project straight into the decoder's embedding space.

• Languages — eight for Aion (English, German, Spanish, Portuguese, Italian, French, Chinese, Japanese, with multilingual data at roughly 6% of training); 140-plus claimed for G​emma.

• Published evaluation — Aion: 11 benchmarks on a named harness, all vendor-run. G​emma: 15-plus benchmarks plus vision, audio and long-context rows, all vendor-run.

What Microsoft's eleven numbers say

The Aion card is more candid than most quiet releases. It states that benchmarks ran on EleutherAI's lm-evaluation-harness where a task implementation existed and on each benchmark's own repository otherwise, that GPT-4o judged the judge-dependent ones, and then it prints the scores: MMLU 55.15, MMLU-Pro 33.35, MMMLU 45.51, GPQA 28.18, GSM8K 74.91, GSMPlus 49.56, MATH-500 48.00, IFEval 83.90, IFBench 32.00, Multi-IF 64.70, chrF++ 43.18.

Two of those carry a qualifier in the notes column — GSM8K is "loose matching", IFBench is "prompt-level loose" — and the card does not hide them. That is the sort of disclosure a marketing team removes and an evaluation team leaves in. It also means the numbers should be read as a floor rather than a pitch: a 33.35 on MMLU-Pro and a 28.18 on GPQA are not figures anyone would choose to lead with.

The card is equally direct about what the model cannot do. Its limitations section says a small model of this size "can miss fine details, provide misstatements, or show inability to follow complex multi-step reasoning", and the recommendations section says the model "was not trained to excel at world knowledge". Those are the vendor's words, not ours, and they line up with the scores rather than contradicting them.

Where G​emma's extra nine billion parameters actually go

G​emma 4 12B's card lists MMLU Pro 77.2, GPQA Diamond 78.8, AIME 2026 without tools 77.5, LiveCodeBench v6 72.0, Codeforces ELO 1659, Tau2 averaged over three runs at 69.0, BigBench Extra Hard 53.0 and MMMLU 83.4. It also prints MMMU Pro 69.1 and MATH-Vision 79.7 for vision, CoVoST 38.5 and FLEURS 0.069 for audio, and 55.3 on RULER at 128K.

Comparing those to Aion's table directly would be dishonest, because almost none of the benchmarks are shared, the harnesses differ, and one is a text-only model being scored against a multimodal one on tasks that reward different training. What can be said is directional and blunt: on the two rows where the benchmark names coincide, MMLU-Pro and MMMLU, G​emma reports 43.8 and 37.9 points more respectively. That is not a scoring quirk. It is what an extra 9.7 billion parameters and seven more months of post-training buy.

The more useful observation is what G​emma does with its size. Its hybrid attention interleaves local sliding-window layers at 1,024 tokens with full global attention, and the global layers share keys and values and apply proportional RoPE — engineering aimed squarely at keeping long-context memory flat. Aion does the same kind of thing at a smaller scale with a 2,048-token sliding window across 21 of its 28 layers. Both models are solving the same problem; one has five times the budget to solve it with.

Footprint is the one column Aion wins outright

At bfloat16, Aion-1.0-Instruct-2.3B is about 4.6 GB. G​emma 4 12B is roughly 24 GB before you touch a quantiser, which puts it on a workstation GPU or a heavily quantised laptop and off a phone entirely. That is not a criticism of G​emma — it was never aimed at a phone — but it is the entire reason Microsoft's model exists. The card's framing is on-device Windows: text and operating-system tasks, on the machine, with the data never leaving it.

The honest caveat is that the 32K window is reached by scaling, not by training at length. The config records original_max_position_embeddings of 2,048 with YaRN factor 16 on the full-attention layers only, and no long-context evaluation of the model has been published. So "32K" is a configured ceiling, not a demonstrated capability. G​emma's 256K has at least a RULER row behind it, vendor-run and at 55.3 — modest, but present.

The factoring decision, not the model decision

These two belong in different layers of the same product, and that is usually how teams end up using them. Aion-1.0-Instruct-2.3B handles the on-device pass: redact, summarise, rewrite, classify, keep the document local. G​emma 4 12B handles the cases that need a screenshot read, an audio clip transcribed, a 100K-token codebase held in context, or a language outside Aion's eight. Neither one replaces the other, and the mistake to avoid is picking the small model for a job that needs the big one because the small one is free to download.

When both are being called from one codebase, the operational cost is not the tokens — it is the second SDK, the second retry policy and the second set of credentials. That is where a router earns its place: OrcaRouter puts 200+ models behind one key at 0% markup over provider list price, with automatic failover and a routing DSL that can send a request down a chain of models rather than to one. To be precise about what is and is not in our catalogue today: the G​emma 4 family is partly present — the 26B A4B and 31B variants are listed — while G​emma 4 12B and Aion-1.0-Instruct-2.3B are not, because the first is a self-hosted weight release and the second has no hosted endpoint anywhere. The routing argument applies to whatever siblings of these models you end up mixing with them, not to these two checkpoints themselves.

A rollout order that fails safely

Start with G​emma 4 12B on the workload you already have, because you can measure it. Its card gives you a denominator, its weights are stable, and any regression you see is your regression rather than an artefact of a three-day-old upload.

Then take Aion-1.0-Instruct-2.3B and run it against the narrow slice you intend to move on-device — say, email thread summarisation or document redaction — using your own data and your own accuracy bar. Do not use its MMLU as a proxy for your task; the model's own card warns that it was not trained for world knowledge, and a 55.15 on MMLU tells you more about trivia than about whether your summaries survive review.

Only then decide whether the local pass replaces a hosted call or sits in front of one. The pattern that works is the boring one: small model does the first pass locally, confidence-gated, with the hard cases escalated to something bigger. That gives you the privacy property on the majority of traffic and keeps the quality ceiling where it needs to be.

A screenshot of the Gemma 4 12B model card, showing its 11.95B parameter count, 256K-token context, text, image and audio input, the MMLU-Pro score of 77.2 and the Apache 2.0 licence.

G​oogle's card prints the numbers Aion's card cannot yet be compared to fairly: MMLU Pro 77.2, GPQA Diamond 78.8, MMMU Pro 69.1 and a 256K window, all vendor-reported.

What would make this a different article

An independent harness run of Aion-1.0-Instruct-2.3B would do it. At the time of writing the repository had three likes and zero recorded downloads, so there is no community evidence at all — no forum threads, no quantised builds, no third-party evals. The card promises quantised versions to follow; when a GGUF lands, the local story stops being theoretical and the download counts start meaning something.

Until then the comparison stands where it is: G​emma 4 12B is a known quantity with a real cost, and Aion-1.0-Instruct-2.3B is a very small, very new, honestly documented model whose most valuable property is that it runs where the network does not reach.

A screenshot of the Hugging Face model card for microsoft/Aion-1.0-Instruct-2.3B, showing the description of the foundational Small Language Model for the Windows ecosystem, the 32,768-token context, the eleven published benchmarks including MMLU-Pro 33.35, and the note that a further optimised version becomes the Windows OS SLM by the end of January 2027.

Microsoft's card is the source for the 2.29B parameter count, the eight supported languages and the 2025-12 knowledge cutoff; its timestamps are the only release signal that exists.

A generated two-column scoreboard comparing Aion-1.0-Instruct-2.3B, with 2.29B parameters, a 32,768-token context, text-only input, roughly 4.6 GB in bfloat16, MMLU-Pro 33.35 and 3 likes on day one, against Gemma 4 12B, with 11.95B parameters, a 256K context, text, image and audio input, roughly 24 GB, MMLU-Pro 77.2 and over a million downloads. Footnoted that both sets are vendor-reported and independently unreproduced, and that the MMLU-Pro rows were not run on a shared harness.

Side by side: the dimension where Aion wins (footprint) and the ones where it does not (context, modalities, languages, and the depth of evidence behind the numbers).