Kolibri vs MathForm-8B: One of These Accuracy Claims Can Be Checked by a Compiler

Kolibri and MathForm-8B share a licence and almost nothing else. Both are Apache 2.0, both are open-weight, both were released in the last three months — Aleph Alpha's Kolibri on 3 October 2026, OpenBMB's MathForm-8B on 14 August 2026 — and both spend a large part of their model cards on mathematics. That is where the resemblance ends. Kolibri is a 78.1-billion-parameter German-and-English mixture-of-experts model that activates 3.46 billion parameters per token and is meant to sit in a regulated document workflow. MathForm-8B is an 8-billion-parameter dense model fine-tuned from Qwen3-8B with one job: take a mathematics problem written in ordinary English and emit a formally correct Lean 4 statement of it. The difference that matters for anyone evaluating either one is not the parameter count. It is that MathForm-8B's accuracy claim is executable. You can check its output with a compiler. Kolibri's cannot be checked by anything but another benchmark run.

That asymmetry is the whole article, and it generalises well beyond these two models. A vendor benchmark table is a claim. A proof assistant accepting a formalisation is a result. When a model's entire output space is something a machine can verify, the marketing layer disappears — either the Lean compiler accepts the statement or it does not, and no amount of launch-post framing changes that.

What MathForm-8B actually produces

Autoformalisation is a narrow, unglamorous and genuinely hard task, and the model card is refreshingly specific about the setup.

• Input and output — a natural-language mathematical statement in, a Lean 4 formalisation out, complete with a theorem header.

• Base model — Qwen/Qwen3-8B, fine-tuned; Apache 2.0 with the rest of OpenBMB's release.

• Training — supervised fine-tuning followed by reinforcement learning on the FormalVerse dataset, with Lean compilation and semantic-consistency feedback driving the reinforcement signal.

• Data pipeline — Mathlib knowledge retrieval, compilation and semantic verification, iterative refinement, then trajectory reconstruction. The pipeline diagram on the model card is the most informative thing in the release.

• Evaluation — Pass@8 pass rates under two separate checks, Syntax Check and Consistency Check, across six benchmarks, reported as an equally weighted macro-average in a figure on the card rather than as a table we can quote row by row.

• Toolchain — a running Kimina Lean Server for the compilation checks, Lean 4.21.0 for the experiments, and a 16,384-token maximum sequence with temperature 0.6 and top-p 0.95.

• Serving — vLLM or SGLang at a 16,384-token context, exposed over an OpenAI-compatible chat interface. That last detail matters more than it looks, because it means the model drops into an existing pipeline as an ordinary endpoint.

Note the two separate checks. Syntax Check is whether the Lean statement parses and type-checks at all. Consistency Check is whether the formal statement means the same thing as the natural-language problem — a far harder property, because a syntactically valid Lean statement that formalises the wrong theorem is worse than a compile error. OpenBMB reports both, which is exactly the right way to do it and is the reason this task has a verification story that general-purpose reasoning does not.

What Kolibri does with mathematics, and why it is a different kind of number

Kolibri is good at mathematics in the benchmark sense. On Aleph Alpha's own post-training harness, at reasoning effort high, it scores 96.9 on AIME 2025 in English and 87.5 in German, 96.0 and 90.0 on AIME 2026, and a 96.5 English average across its mathematics suite against 88.8 in German. For context on the same table, Kolibri's 96.9 on AIME 2025 English sits above Nemotron 3 Super 120B-A12B at 91.7 and Qwen3.6 35B-A3B at 84.6, and just under Qwen3.8 27B at 97.9.

Every one of those figures is vendor-reported, on a vendor harness, with no independent reproduction, and there is no Artificial Analysis page for Kolibri to cross-check against. That is not a criticism of the numbers; it is a statement about what kind of object they are. An AIME score is a percentage of correct final answers on a multiple-choice examination. It tells you the model can reach an integer. It tells you nothing about whether the reasoning that produced it was sound, and there is no artifact left behind that a third party can inspect.

Put next to MathForm-8B's output, that difference is stark. A Kolibri answer to an AIME problem is a number. A MathForm-8B output is a Lean 4 theorem statement that either compiles against Mathlib or does not. If you are building a system where a mathematical claim has to be defensible — a formal-verification pipeline, a proof-assistant workflow, an audit trail — the second artifact is worth considerably more than the first, and no benchmark row expresses that.

Where the two would actually meet

Framed as a fight, this matchup is uninteresting: an 8B specialist beats a 78B generalist at formalising mathematics and loses at everything else, including German administrative prose, long-context document reasoning and tool-calling across a hundred-step agent trajectory. But the two are not substitutes, and the useful question is what a pipeline built from both looks like.

The natural composition is a routing one. A generalist with strong reasoning and tool-calling handles the ingestion, the disambiguation and the retrieval; a specialist is invoked for the two per cent of cases that need a formal artifact. Doing that by hand means two vendors, two contracts, two SDKs, two sets of credentials and a dispatch layer that somebody maintains. It is the case where a single endpoint earns its keep: one OpenAI-compatible key, a routing rule that sends the mathematics-shaped requests to the formaliser endpoint and everything else to the generalist, and failover when one of them is slow. That is the routing DSL's job — composing several models into one call rather than hard-wiring a choice at development time — and where a panel of models answering together is useful, model fusion covers it. Neither Kolibri nor MathForm-8B is on OrcaRouter today; we probed the catalogue for both under every vendor and model spelling and neither is there. The composition argument is about the shape of the problem, not about these two specific endpoints.

What is on the catalogue is the generalist half of that pattern at a price you can measure. Qwen3.8-27B is listed at $0.33 per million input tokens and $2.40 output with a 262,144-token window, and Qwen3.8-Max at $2.00 and $6.00 with a 1M-token window. For a team exploring whether a formalisation step is worth adding to a document pipeline at all, the cheap experiment is to route the generalist work there, measure the volume of requests that genuinely need a Lean artifact, and only then decide whether a 16,384-token specialist endpoint is worth provisioning. Provider list price is passed through with nothing added per token, so the numbers move the day a vendor moves them.

Licence is the one line where they are identical

Both are Apache 2.0, and in a category where bespoke research licences and acceptable-use riders are common, that is a genuine point of parity worth naming — it means neither model requires a legal review before it can be used commercially, modified, or redistributed.

The obligations diverge elsewhere. Kolibri brings a ~78 GB weights footprint and a hardware floor of two A100 80 GB cards, two H100 SXM5s, one H200, one B200 or one B300, plus the vendor's aleph-alpha-inference package and vLLM plugin. MathForm-8B in bfloat16 is roughly 16 GB of weights and serves on a single modern accelerator at a 16,384-token context; its dependencies are a Lean toolchain and, for the evaluation pipeline, a running Kimina Lean Server. One of those deployments fits in a workstation. The other does not.

The context figures cut the other way and by a wide margin. Kolibri's native window is 262,144 tokens, validated to 1,048,576, which is what makes it a document model: a full German regulatory filing or an aerospace maintenance manual fits in one call. MathForm-8B is capped at 16,384 tokens by design, because a formalisation request is a single problem statement and there is no reason for it to be longer. Neither number is a deficiency. They simply describe different jobs.

Choosing, and the verification question underneath

• Pick MathForm-8B if your output has to be checkable. If a downstream system consumes Lean 4, or if the whole point is that a proof assistant signs off on the result, no generalist substitutes for it, and the Pass@8 numbers under Syntax Check and Consistency Check are the ones to interrogate rather than any AIME row.

• Pick Kolibri if you need one model that reads German and English documents at long context, reasons across them, calls tools, abstains when the context does not support an answer, and can be deployed inside your own perimeter under a licence you can state in one line. Mathematics is a capability it has, not a product it is.

• Consider both if you are building a formalisation pipeline. Not as alternatives, but as two endpoints behind one routing rule, with the specialist called for the narrow slice of requests that need it.

And if you are evaluating either on the strength of a benchmark number, apply one test first: ask what artifact the number leaves behind. For MathForm-8B there is a Lean file and a compiler that either accepts it or does not, and you can run both yourself this afternoon. For Kolibri there is a percentage in a launch table, vendor-reported, unreproduced, with no independent index page to check it against — and the only way to falsify it is to download 78 GB of weights, rent the hardware, and re-run the harness. That asymmetry is worth more than the score itself when you are deciding what to put in production.