What Is MathForm-8B? OpenBMB's Quiet Autoformalization Release Turns Math into Lean 4

openbmb/MathForm-8B is a new autoformalization model from OpenBMB that translates natural-language mathematical statements into Lean 4, and it shipped with almost no announcement: the weights, the dataset, and the paper all appeared on Hugging Face and arXiv on the same day, 2026-08-14, under the umbrella title "MathForm: Scaling Mathematical Autoformalization with Knowledge Retrieval and Verification-Guided Refinement." That quiet launch hides an unusual result — an 8B-parameter model reporting average Pass@8 scores of 88.06% under a syntax check and 72.37% under a stricter consistency check across six benchmarks, which the paper claims beats several specialized 32B autoformalizers. This is a what-we-know-so-far piece: everything below labeled "from the repo" comes straight from the model card, the dataset card, and the paper, and anything not yet independently confirmed is marked as such.

Key takeaways

• MathForm-8B is an 8B, Apache-2.0 autoformalization model: it reads an informal math problem and writes a Lean 4 theorem statement with a named header, ready for a proof later.

• It is fine-tuned from Qwen3-8B on FormalVerse, a ~367,000-example verified Lean 4 dataset OpenBMB built with knowledge retrieval and compiler-checked refinement, then trained with reinforcement learning using Lean compilation and semantic-consistency feedback.

• Reported numbers (vendor-reported, unreproduced): 88.06% average Pass@8 under Syntax Check, 72.37% under Consistency Check, beating 7B-to-32B specialized autoformalizers in the paper's own table.

• It is not announced, not on a major paid API at launch, and not yet independently benchmarked — three gaps that matter for production adoption.

• Serving is self-hosted: Transformers, vLLM, or SGLang, all exposing an OpenAI-compatible endpoint.

What the release actually contains

Three artifacts went up within minutes of each other on 2026-08-14, which is what a coordinated-but-unannounced release looks like:

• The model repo, openbmb/MathForm-8B — an 8B causal LM in BF16 with a chat template, four safetensors shards, Apache 2.0 license.

• The dataset repo, openbmb/FormalVerse — a Lean 4 autoformalization dataset of roughly 367,000 verified examples, also Apache 2.0.

• The paper, arXiv 2608.14221 — 25 pages describing the data-construction pipeline, the training recipe, and the six-benchmark evaluation.

The GitHub code link in the README is still a placeholder at the time of writing, so the evaluation pipeline and Pass@k scripts are promised but not yet public. The README does say compilation checks require a running Kimina Lean Server and that experiments use Lean 4.21.0.

The repository page above is the whole public surface of the release right now: a model card, four safetensors shards, a chat template, and a README that doubles as the only documentation. No announcement blog post exists at the time of writing.

What MathForm-8B does — and why it is a narrow job

Autoformalization is the step before theorem proving: given a math problem in plain English ("Show that for every real number x, x² is non-negative"), the model must produce a formally correct statement in Lean 4 — imports, types, and a theorem header — that a human or a prover can then attack. It is a genuinely different skill from doing the math, because the model has to map natural-language concepts onto Mathlib's exact hierarchy of definitions and types. A statement that type-checks but silently weakens the original ("(2^5) ∣ (13^4 − 11^4)" instead of the full divisibility claim) is the classic failure mode, and it is why the paper distinguishes Syntax Check (does it compile) from Consistency Check (is it semantically the same statement).

The model card shows the intended usage pattern: you hand it a prompt with the informal problem and a desired theorem name, and it returns a Lean 4 statement with code>theorem my_favorite_theorem : ... := by sorry/code> — the code>sorry/code> leaves the proof obligation open. That division of labor matters: MathForm-8B is a formalizer, not a prover. Teams building Lean tooling use it to convert problem banks into machine-checkable form.

How it was trained

The paper's recipe is two-stage. First, OpenBMB built FormalVerse with a pipeline that (1) retrieves relevant definitions and existing formalizations from Mathlib before generation, (2) generates candidate statements, (3) refines them using Lean compiler diagnostics and semantic-consistency feedback, and (4) keeps only samples that pass both checks. That verified corpus is then used for supervised fine-tuning, followed by reinforcement learning with reward signals from Lean compilation and semantic consistency.

The dataset card gives a concrete flavor of the data: each entry pairs an informal statement with a verified formal one, tagged by source (e.g., AceReason-Math) and topic label (Number Theory, and so on). Because every example passed a real compiler check before it entered training, the model learns from statements that are known-good rather than from a model's raw output.

The benchmark table, honestly labeled

All numbers in this section are vendor-reported from the paper (arXiv 2608.14221) and have not been independently reproduced. Pass@8 means the model gets eight tries per problem and the run counts if any one passes; this is a friendlier metric than pass@1 and should be read as "how often the model can produce a correct statement given budget."

• MathForm-8B averages — Syntax Check 88.06%, Consistency Check 72.37%.

• Per benchmark, SC then CC: FormalIMATH 100.00 / 95.06, ProverBench 100.00 / 94.83, CombiBench 93.00 / 47.00, FATE-M 99.33 / 97.33, FATE-H 82.00 / 63.00, FATE-X 54.00 / 37.00.

• The hard sets are the honest ones: FATE-H CC 63% and FATE-X CC 37% show the model's ceiling on the hardest subsets, versus 95%+ CC on the easier FormalIMATH and ProverBench.

• The best 8B baselines the paper lists — ReForm-8B 81.76 / 66.21, Goedel-Formalizer-V2-8B 78.24 / 60.08 — and the best 32B baselines — ReForm-32B 81.61 / 68.41, Goedel-Formalizer-V2-32B 78.28 / 63.74, StepFun-Formalizer-32B 63.65 / 44.47 — all trail MathForm-8B's 88.06 / 72.37.

• The SFT-only checkpoint (before the RL stage) lands at 84.38 / 66.53, so the reinforcement-learning pass is worth roughly +3.7 SC and +5.8 CC on average, with the biggest gains on the hard sets.

The strongest claims to be skeptical about: the 100.00 SC scores on FormalIMATH and ProverBench (100% compile on the easy sets is a red flag that those sets have converged), and the comparison against 32B models that were not re-run under identical conditions. The Consistency Check numbers on FATE-H and FATE-X are the figures most likely to survive independent testing.

What is not confirmed

• No independent evaluation exists. No third party has run MathForm-8B through a public harness as of writing, and the eval code has not shipped.

• No serving announcement. OpenBMB has not posted a launch blog, a pricing page, or an API endpoint. The "quietly shipped" framing is literal.

• The RL reward weights, training budget, and hardware are not in the model card; they live in the paper only.

• Whether the 8B model generalizes to Lean 4.21.1+ or to non-Mathlib imports is untested.

How to run it

Self-hosting is the only route today. The README documents three paths, all with an OpenAI-compatible chat endpoint at code>localhost:8000/v1/chat/completions/code>:

• Transformers — code>AutoModelForCausalLM.from_pretrained("openbmb/MathForm-8B", torch_dtype=torch.bfloat16)/code>, then generate with the chat template.

• vLLM — code>vllm serve openbmb/MathForm-8B --served-model-name MathForm-8B --dtype bfloat16 --max-model-len 16384/code>.

• SGLang — code>python -m sglang.launch_server --model-path openbmb/MathForm-8B --served-model-name MathForm-8B --dtype bfloat16 --context-length 16384/code>.

The README recommends temperature 0.6, top_p 0.95, and up to 16384 new tokens — formal statements run long, so the generous generation window is the actual system requirement to budget for.

Why the "8B beats 32B" part matters

If the numbers hold up, MathForm-8B is the strongest argument yet that the autoformalization bottleneck is data quality and verification, not raw parameter count. The paper's own table shows 32B specialized models (ReForm-32B, Goedel-Formalizer-V2-32B, StepFun-Formalizer-32B) sitting below an 8B model trained on a compiler-checked corpus. For teams that currently run a 32B formalizer, that is a material cost change — an 8B model at BF16 fits in a single GPU that most 32B models cannot, and it serves faster per token.

It also sets up the honest choice that the rest of the model landscape keeps producing: a narrow specialist that does one verified task very well, versus a general model that can attempt many tasks with no verification guarantee. For formalization specifically, the specialist is the one with a compiler checking its output — which is precisely the property that makes a router with automatic failover comfortable to put in front of it. A routing layer like the one OrcaRouter runs across 200+ models, at provider-list-price pass-through, lets you point a test path at a days-old open-weights model like this and fall back to a proven model the moment it stalls — you can adopt a quiet release without betting your production path on it, and there is no markup on the token price if a provider later lists it.

What to watch next

The three things that would turn this from "interesting repo" into "trusted tool": the GitHub evaluation code actually appearing; a first independent pass at FATE-H and FATE-X under pass@1 instead of pass@8; and any OpenBMB announcement that adds a hosted route or a paper v2 with ablation numbers. Until at least one of those lands, treat the headline scores as directional — the architecture and the training-data idea are the durable news, not the exact percentage.