Beam 1.0 vs MathForm 8B: One Model's Errors Are Caught by a Compiler

Ask MathForm 8B to formalise a theorem and the answer is either a Lean 4 statement that compiles or one that does not. Ask Beam 1.0 to fix a bug and the answer is a diff that runs or one that does not, and the distance between those two things is the entire difference between these models. MathForm 8B is OpenBMB's 8.19-billion-parameter autoformalization model, built on A​libaba's Qwen3-8B and trained on FormalVerse, a corpus of roughly 367,000 verified Lean 4 examples, through supervised fine-tuning followed by reinforcement learning on Lean compilation and semantic-consistency feedback; it was published on 14 August 2026 under Apache 2.0. Beam 1.0 is Reflection's 501.3-billion-parameter sparse mixture-of-experts model with 23.4 billion active per token, announced on 5 October 2026 and shipped as downloadable weights on 16 October 2026, also under Apache 2.0. The size ratio is 61 to one and it is the least interesting number on the page.

The interesting one is that MathForm 8B produces output a machine can adjudicate without trusting anyone, and Beam 1.0 does not. Everything below follows from that.

What each model is asked to produce

MathForm 8B has one output contract. It takes a mathematical statement written in ordinary prose and returns a Lean 4 statement — a header, a named theorem, the hypotheses and the goal. The training pipeline is built around that shape: Mathlib knowledge retrieval to ground the statement in existing definitions, compilation and semantic verification to reject bad candidates, iterative refinement to repair them, and then trajectory reconstruction to turn the surviving examples into training data. The paper reports the model at 88.06% average Pass@8 under syntax checking and 72.37% under consistency checking across six benchmarks. Those are OpenBMB's numbers, unreproduced by anyone else, and weighted toward the easier sets in the evaluation — but they are numbers about a task that has a right answer.

Beam 1.0 has no such contract. It is a general model: its card describes the chat template, the Harmony-style wire format with analysis and final channels, six levels of reasoning effort mapping to token budgets of 8, 32, 64, 128, 256 and 512, and a tool-calling convention where functions are declared as flat TypeScript and invoked in XML bodies. Its evaluation covers agentic coding (SWE-bench Verified 81.6% on BF16, Terminal Bench v2.1 80.1, SWE Bench Pro v1 66.1, DeepSWE v1.1 43.3), reasoning (GPQA Diamond 90.6, AIME 2026 97.7, LiveCodeBench 88.3), tool use (MCP Atlas 80.8) and general capability (LongBench v2 67.5, IFBench 79.6). Every figure is Reflection's own. Whether the code it writes is *correct* is established by running it, reviewing it, or shipping it and finding out.

So: one model's failures are self-announcing. The other's are ordinary software failures that arrive later, in review or in production, and look like every other bug you have ever had.

The benchmarks do not overlap, and that is the finding

Line up the two evaluation sets and there is almost nothing to subtract.

• What Beam 1.0 is scored on — repository-scale issue resolution, terminal tasks, competitive programming, graduate-level science questions, tool orchestration, long-context recall and instruction following.

• What MathForm 8B is scored on — Pass@8 formalisation accuracy under a syntax checker and under a semantic-consistency checker, across six mathematical benchmarks, against the paper's 7B-to-32B specialised baselines.

• The overlap — essentially zero. There is no row where both models were measured by the same party on the same task, and there will not be, because one is measured on general capability and the other on a single verifiable transformation.

• The one shared dimension that does exist conceptually — reasoning — is measured on different populations. Beam 1.0's 90.6 on GPQA Diamond is a multiple-choice science score. MathForm 8B's 72.37% consistency rate is a formalisation check. Neither predicts the other.

This is why a size comparison here would be misleading rather than merely unflattering. Beam 1.0 is sixty-one times the model, and it cannot do MathForm 8B's job at all without being trained to — the output contract is narrow enough that general capability is not a substitute for it. A 501B generalist asked to produce a named Lean theorem with a correct header will produce something plausible-looking and frequently wrong, which for a proof assistant is the worst possible failure mode.

What you need to have in order to run either one

MathForm 8B inherits Qwen3-8B's licence, which the Hub records as Apache 2.0, and this is the practical case for it: 8.19 billion dense parameters, an English-only model, a narrowing of the generation budget to Lean output rather than general reasoning, and a download rather than a cluster. Its base, Qwen3-8B, is what a reader would otherwise self-host — a general 8B with a 32K native context extensible to 131K and coverage of 119 languages. MathForm 8B trades nearly all of that away for one skill.

Beam 1.0 needs somewhere to live. The model card gives 52 layers, 256 experts with eight routed per token, a 152,000-token vocabulary, a knowledge cutoff of 30 June 2026 and a 1,000,000-token context window. It ships in three weights — BF16 at 501.3B, FP8 at 501.3B, NVFP4 at 255B — plus a 2B DFlash2 drafter for speculative decoding. There is no serving recommendation on the card and no published API price anywhere on Reflection's properties; Artificial Analysis's page for the model lists the price fields as unavailable and notes that no publicly available API exists. The four-bit build is what makes this a plausible purchase rather than a hypothetical one, and it is still a GPU conversation.

OrcaRouter routes neither model, and this page does not imply otherwise. What we route is the served tier those two sit outside of: more than 200 models behind a single OpenAI-compatible key at provider list price with 0% markup, so a vendor price cut reaches your bill the day it is announced. For a team whose real need is a model that answers today rather than a checkpoint that compiles tomorrow, that is the row this comparison cannot supply.

The verifiability difference, in plain terms

Here is the asymmetry stated as an operational fact rather than a benchmark.

A MathForm 8B output enters a pipeline that has a gate. The Lean compiler either accepts the statement or rejects it. Semantic consistency checking either passes or fails. A bad output can be discarded automatically, and the model can be left in an agent loop that retries until something verifies — which is exactly what the RL stage trained it to do. The cost of a wrong answer is a rejected sample, not a shipped defect. That is a genuinely rare property in the current open-weights field, and it is why a specialised 8B model can be more useful than a general one a hundred times its size inside a pipeline shaped like formal mathematics.

Beam 1.0's outputs have no such gate. The evaluation table can tell you how often it resolved a GitHub issue on SWE-bench, but it cannot tell you whether the patch it writes for your repository is correct, because that is a question about your tests. The model card is unusually specific about the wire format and unusually silent about the accuracy question, and that is a fair description of where the field is: the tooling around agentic coding models is good at executing code and bad at certifying it.

If your domain has a checker — a type system, a proof assistant, a schema, a simulator — a specialist aimed at that checker is often the higher-leverage download. If it does not, you are buying capability and paying for evaluation yourself, and the larger general model is the one that gives you more of it.

Choosing between them

Pick MathForm 8B if the work is formalisation and nothing else — turning a corpus of mathematical prose into Lean 4 that a proof assistant will accept, at a scale where a human cannot check each one. The 8.19-billion-parameter footprint, Apache 2.0 terms and narrow output shape are all advantages there. Do not pick it for anything else: it does not chat, it does not write prose worth shipping, it does not handle 119 languages the way its base model does, and it has no documented capability outside the one transformation it was trained for.

Pick Beam 1.0 if the work is the general agentic kind — resolving issues in large repositories, driving tools, executing long multi-step plans — and if you have the hardware or a serving partner for a 501B MoE. The 1M-token context, the published evaluation table and the four-bit checkpoint are all aimed at that workload. Budget for the reasoning tokens: the template has no off switch, the served default is the maximum effort level, and stripping prior-turn reasoning is documented as degrading output.

Pick both if you are building the pipeline this article is really about. The pattern that makes sense is a general model to do the ambiguous work and a verifiable specialist to do the part with a gate — and the two roles fail in different ways, which is the only reason it is worth owning both.

Where this comparison will look different in three months

Two rows are missing and both are datable. The first is an independent evaluation of Beam 1.0's weights, which became possible on 16 October 2026 and has not happened yet; every figure this page attributes to Reflection is still single-source. The second is Beam's price. A model with a 501-billion-parameter evaluation table and no metered rate is a model most teams cannot budget for, and until Reflection or a serving partner publishes one, the honest cost comparison between these two is a download against a rental that has not been quoted.

MathForm 8B's gap is different and will not close by waiting. Its reported 88.06% syntax-check and 72.37% consistency figures are a vendor's numbers on a task with a verifier, and the verification is the product — anyone with the model and a Lean installation can measure it themselves, which is more than can be said for most of the numbers on this page. That is what a narrow contract buys you.

None of that is hypothetical: the documented AI failures and agent incidents behind releases like this one are collected in the OrcaRouter AI Incident Archive, where each entry carries its own sources, severity, confidence grade and disputed flag.