Ember-1 vs MathForm-8B: Two Models Built by Narrowing a Borrowed Base

Ember-1 and MathForm-8B share a strategy that neither lab advertises as one: both are narrowings of a model somebody else trained. Ember-1 is Fireworks Research's specialized derivative of Moonshot AI's Kimi K3, published 23 September 2026, retrained so it reaches K3's accuracy on roughly 40% fewer tokens. MathForm-8B is OpenBMB's 8B autoformalization model, released quietly on 14 August 2026 as an Apache-2.0 fine-tune of Aliba​ba's Qwen3-8B that turns informal mathematics into Lean 4 theorem statements a compiler can check. One narrowing removed wasted deliberation and kept general capability intact. The other removed almost all general capability and bought verifiability instead. Putting them together is the cleanest way to see what a specialization actually costs, because the two models spent their training budgets on opposite sides of that ledger.

Two kinds of narrowing

Fireworks Research's intervention is behavioural. Ember-1 keeps Kimi K3's architecture and its breadth — mathematics, coding, instruction following, conversation, search, tool use and software engineering all appear in the training mix — and changes only how long the model deliberates before answering. The reported result is that reasoning length fell by 35–50% without accuracy loss across seven benchmarks and two customer production A/B tests, with one production coding workload dropping from 49.3K to 29.9K output tokens while its score held at 0.753 against 0.751. Every figure is vendor-reported and unreproduced.

OpenBMB's intervention is contractual. MathForm-8B takes Qwen3-8B and points the entire training budget at one output shape: a Lean 4 statement with an imports header and a named theorem. The pipeline is supervised fine-tuning on FormalVerse — a corpus of roughly 367,000 verified Lean 4 examples OpenBMB built and released alongside the model — followed by reinforcement learning that uses Lean compilation and semantic-consistency feedback as the reward signal. The model does not solve proofs. It writes the statement a prover will finish, and the paper's own framing describes the six-benchmark evaluation as the point of the exercise.

What each one gave up

Ember-1 gave up very little on paper, which is the whole claim. Its published sheet shows a win on Terminal Bench 2.1 at 82.0% against Kimi K3 Max's 80.9% and on DeepSWE 1.1 at 75.2% against 66.4%, and narrow losses on SWE-bench Verified at 92.2% against 93.2% and SWE-Interact at 20.0% against 21.3%. Those are vendor numbers on vendor-chosen sets, but the shape is consistent: a model that has not lost ability so much as redirected where it spends effort. The token savings, though, range from 51.9% on Terminal Bench down to 5.9% on τ-2 Bench Airline, so "about 40%" is an average across a very wide spread.

MathForm-8B gave up most of what Qwen3-8B is known for. It does not hold a general conversation, does not cover the 119 languages and dialects Qwen3-8B was trained on, and does not accept images or audio. Its generation budget is sized for Lean output, not for extended mixed reasoning. What it kept is a permissive license and a small footprint: four safetensors shards in BF16, running under Transformers, vLLM or SGLang behind an OpenAI-compatible endpoint, with a compilation path that expects a Kimina Lean Server on Lean 4.21.0.

The numbers measure different things, and the gap is the point

Ember-1's headline figure is a percentage of tasks completed correctly by an agent — Terminal Bench 2.1, 89 samples, 82.0%. MathForm-8B's headline figures are average Pass@8 scores across six autoformalization benchmarks: 88.06% under a syntax check and 72.37% under a stricter consistency check. Those are not on the same axis. One measures whether an agent finished a job in a terminal; the other measures whether a generated theorem statement parses and whether it means the same thing as the informal problem it came from.

The 88.06 against 72.37 spread inside MathForm's own results is the more instructive number. The gap between "this compiles" and "this compiles and says what I meant" is roughly sixteen points, and it is the failure mode that makes autoformalization hard: a statement that type-checks while quietly weakening the original claim is worse than an obvious error, because nothing downstream flags it. On the hardest sets the consistency check drops to 63% on FATE-H and 37% on FATE-X, while easy sets like FormalIMATH sit at 95.06% and ProverBench at 94.83%. That is a specialist being honest about where a specialist is weak, and it is more useful than a single average.

The contrast, dimension by dimension

• Base model — Ember-1: Kimi K3. MathForm-8B: Qwen3-8B.

• What the training changed — Ember-1: how long the model reasons, with capability held constant. MathForm-8B: what the model outputs, with generality largely surrendered.

• Parameters — Ember-1: undisclosed. MathForm-8B: ~8B, dense, BF16.

• Output contract — Ember-1: ordinary text and tool calls, at K3-level quality. MathForm-8B: a Lean 4 statement with a header and a named theorem.

• License and weights — Ember-1: none published; research preview through the vendor's own platform. MathForm-8B: Apache 2.0, weights and dataset both downloadable.

• Reported headline — Ember-1: 82.0% Terminal Bench 2.1 with 51.9% fewer tokens. MathForm-8B: 88.06% average Pass@8 under syntax check, 72.37% under consistency check.

• Independent verification — neither; both are vendor-reported and unreproduced.

The license line decides more than the benchmarks do

For all the numerical difference, the practical difference between these two releases is distribution. MathForm-8B is a file. OpenBMB published the weights, the FormalVerse dataset and the paper on the same day, under Apache 2.0, with no announcement and no hosted API — the model card is the launch. You can download it this afternoon and run it on a single GPU, and nobody can take it back. Ember-1 is a service. There are no weights, no published price, and the access window is described as a two-week serverless period whose continuation depends on demand. You can call it today and you cannot be certain you can call it in November.

That difference also sets what each model can be used for. A formalization component belongs inside a pipeline you control, pinned to a version, with the Lean toolchain on the same machine — which is why an ungated Apache-2.0 checkpoint is the right shape for MathForm-8B's job, and why the missing GitHub code link on its README (still a placeholder at time of writing) is a more annoying gap than any benchmark number. A reasoning-cost model belongs behind an API, where the token bill is the thing being optimized, and where vendors compete on price and latency. Ember-1's shape fits its job too; it just means the dependency is commercial rather than technical.

Where a pipeline would use both

These two models are complementary rather than competing, and the composition is easy to describe: a formalization specialist converts a problem into a checkable statement, and a reasoning model works on the statement or the surrounding engineering. Neither is on OrcaRouter — MathForm-8B is self-hosted only, and Ember-1 is on the vendor's own preview — but the composition itself is a pattern our routing DSL exists for. Composing several models into a single call is how a pipeline gets a specialist and a generalist without maintaining two integration paths and two contracts, and model fusion goes one step further by letting a panel of models answer together when a single one's failure mode is expensive.

For a formalization stack specifically, the case for composition is stronger than usual. The visible failure mode is a statement that compiles and means something slightly different, and the cheapest defence against a silent error is a second model reading the same problem — which is a routing decision, not a training decision.

Which is the better buy

If you need machine-checkable mathematics, MathForm-8B is the only one of the two that produces any, and its main cost is the generality you were not going to use for this task anyway. Download it, budget for the Lean server, and build your own evaluation — OpenBMB's paper will not tell you how it performs on your distribution.

If you need a general reasoner with a smaller token bill, Ember-1 is aimed at you, and the right next step is shadow traffic against whatever you run today rather than a benchmark comparison. Its risk is availability, not ability, and that is a risk you can hedge by keeping the routing layer between your application and the model.

The uncomfortable conclusion for anyone hoping one of these settles the question is that neither has been independently evaluated. MathForm-8B has been public for six weeks and no third party has published a reproduction; Ember-1 has been public for a day. Both ask you to be the evaluator, which is the normal condition of picking a specialized model in 2026.

What they do prove is that the narrowing strategy works in both directions. A frontier model can be made cheaper without being made worse, and a small base model can be made rigorous by pointing its training at a compiler. The interesting question is not which of these two approaches wins, but how much longer either remains necessary once the techniques in them become standard practice.