HiRO-ACEv2 vs MathForm 8B: Two Narrow Models, Two Kinds of Ground Truth

Neither HiRO-ACEv2 nor MathForm-8B is trying to be a general model, and both were built by teams who decided the interesting problem was narrower than a chatbot. HiRO-ACEv2 is Ai2's second-generation climate emulator, posted to Hugging Face on 12 October 2026 as a 10.9 GB repository with no announcement, alongside a preprint dated the following day. MathForm-8B is OpenBMB's autoformalization model, created on 14 August 2026 on a Qwen3-8B backbone, that turns natural-language mathematics into Lean 4 statements. One produces a 3 km gridded atmosphere; the other produces a formal proof obligation. The reason to compare them is not capability — they have no overlapping use — but the thing they both claim that turns out to mean something very different in each case: that a machine can tell whether the output is right.

What each model actually emits

HiRO-ACEv2 runs a two-stage pipeline. An autoregressive emulator generates X-SHIELD, the global storm-resolving model, coarsened to 100 km; a separately trained diffusion model then downscales that coarse state to 3 km, producing six-hourly averaged surface precipitation rates, instantaneous 10-metre eastward and northward winds, and sea level pressure. The repository holds the downscaler as HiROv2.ckpt at about 1.28 GiB and the coarse emulator as ACE2S-X-SHIELD.ckpt at about 2.98 GiB, plus ten years of NetCDF forcing data, two initial-condition files and the YAML configs that tie them together.

MathForm-8B emits text, but a very particular kind: a Lean 4 theorem statement. The card describes it as an autoformalization model trained on OpenBMB's FormalVerse dataset, first with supervised fine-tuning and then with reinforcement learning whose reward comes from Lean compilation and semantic-consistency feedback. It is 8B dense, Apache 2.0, and it inherits its scale from its base — Qwen3-8B — rather than from any published parameter count of its own. Recommended generation settings are temperature 0.6, top-p 0.95 and 16,384 new tokens, on a Transformers, vLLM or SGLang server; evaluation needs a running Kimina Lean Server and Lean 4.21.0.

Both are narrow by design. That is the first genuine similarity, and it is where the similarities mostly stop.

Two kinds of check, and why only one of them settles an argument

The Lean side has a decisive verifier. A theorem statement either compiles under Lean 4 or it does not, and the type checker is a fixed calculus — it does not have good days and bad days, and it does not need to be calibrated. That is why MathForm-8B's training loop can use compilation as a reward signal at all. The published figures are Pass@8 rates under two regimes, syntax checking and consistency checking, evaluated across six benchmarks; the syntax-checked figure sits at 88.06% and the consistency-checked figure at 72.37%. Those are the authors' own numbers, unreproduced here, and the sixteen-point gap between them is the most informative thing on the card: a statement that compiles is not the same as a statement that says what the original problem meant.

The climate side has no such verifier. HiRO-ACEv2's model card contains no benchmark table. Its one quantitative claim is that the downscaler reproduces the reference precipitation, 10-metre wind and sea level pressure distributions out to the 99.9999th percentile, and that it broadly follows the reference tropical cyclone intensity distribution. The paper adds that inference costs more than 300 times less than a conventional numerical global storm-resolving model.

Every one of those is a distributional claim against a reference, not a pass-fail test against a calculus. That distinction decides how long each claim survives unchallenged. You can falsify the MathForm-8B number by running the model against the benchmarks with a Lean server and a GPU. Falsifying the HiRO-ACEv2 number requires a ten-year forcing dataset, a reference simulation to compare against, and an analyst willing to work in the extreme tail. The first is a week's work for a graduate student. The second is a paper.

What the vendors will and will not tell you

The two model cards have opposite temperaments, and reading them side by side is instructive about what a card is for.

OpenBMB's card is a claim sheet with a limitations section. It gives the training corpus, the reward structure, the recommended sampling parameters, the exact Lean version, and a figure reporting Pass@8 across six benchmarks — though notably the numbers live in an image rather than in the text, so anyone scraping the page programmatically will find the method described carefully and the results not at all. What it does not give is a release date; the only date signal anywhere in the metadata is the arXiv identifier 2608.14221 and the repository's creation timestamp of 14 August 2026.

Ai2's card is essentially a list of constraints. Nine years of training data, 2014 to 2022, for both checkpoints. Outputs that reproduce X-SHIELD rather than observations, so they carry the reference simulation's biases. No expectation of generalisation outside the training forcing conditions — a point the v2 paper sharpens by noting the framework is untested beyond 66°S to 70°N and cannot run past 88°S or 88°N. Patch-overlap discontinuities in regions larger than the 16-by-16 one-degree training tile. A note that the bundled ACE2S-X-SHIELD checkpoint is less rigorously tested than the earlier ACE2-ERA5 line, and that it is not for weather forecasting. And a licence — Apache 2.0 — carrying Ai2's Responsible Use Guidelines and a research-and-educational framing.

One card tells you what the model scores. The other tells you what it cannot do. Neither is hiding anything, and a reader who mistakes the second for a weaker release has misread what these artifacts are for.

The part that is genuinely comparable: what a fork is worth

Here is the one axis on which the two releases can be measured the same way, and it goes in opposite directions.

• Where the backbone comes from — MathForm-8B is a full fine-tune of Qwen3-8B, so its lineage is documented in the Hugging Face metadata and its ancestor is a widely deployed open model; HiRO-ACEv2's components are trained within Ai2's own pipeline, with training data attributed to X-SHIELD, provided by GFDL, and no upstream open checkpoint underneath.

• How many runtimes can load it — MathForm-8B runs on Transformers, vLLM or SGLang, and the community has already produced GGUF and MLX quantisations for llama.cpp and Apple silicon; HiRO-ACEv2 runs through Ai2's fme Python package with configs pointing at the ai2cm/ace repository, and there is no alternative path.

• How fast the ecosystem moved — within four days of MathForm-8B's repository appearing, community GGUF builds were online, and 4-bit, 6-bit and 8-bit MLX conversions followed within two weeks; HiRO-ACEv2's repository has a single like and no downloads two days after its last commit.

• What the weights cost you to obtain — MathForm-8B downloads as a normal model checkpoint and quantises down to consumer hardware; HiRO-ACEv2 is 10.9 GB of repository, most of it forcing and initial-condition data rather than weights, and it does not shrink.

The pattern is that an application-specific language model inherits an ecosystem from its backbone, and an application-specific physical emulator inherits nothing. For MathForm-8B, the work of making it run was mostly done before it existed. For HiRO-ACEv2 it starts at the README.

Where a routing layer fits, and where it does not

The honest answer here is asymmetric, and it is worth stating plainly rather than gesturing at both models equally. MathForm-8B itself is not in our catalogue — the fine-tuned checkpoint comes from OpenBMB's own distribution and needs to run on your own server next to the Lean installation it calls. Its ancestor is not on our catalogue either, which is worth saying because the assumption runs the other way: the specific Qwen3-8B checkpoint it was fine-tuned from is a 2025 release and is not one of the Q​wen ids we serve. What we do route is the current Q​wen generation around it — the Qwen3.8 and Qwen3.5 tiers, and the Qwen3-VL-8B-Instruct checkpoint that shares the 8B scale and the same family lineage at $0.18 per million input tokens and $0.70 per million output over a 131,072-token context.

That is the layer a formalization pipeline actually needs. Splitting a textbook into candidate statements, extracting the informal theorem out of a paper before it goes to the autoformalizer, or turning a Lean error message into a diagnosis are ordinary language-model calls, and they can run through the same key as everything else. HiRO-ACEv2 is not routeable and never will be in its current form, because there is no token-metered request to route — 10.9 gigabytes of checkpoints and a shell script is not an endpoint.

What one key across two hundred-plus routed models buys you, in a pipeline built around either of these, is that the surrounding work becomes a call instead of a project: provider list price passes through with no markup added, failover between providers is automatic, and a script that drives an ensemble or a formalization run does not need a second vendor contract to summarise its own output.

What to watch for on either side

A number that matters on the Lean side would be an independent reproduction of the 88.06 and 72.37 figures by a group with no stake in the training recipe, ideally disaggregated per benchmark rather than averaged into a single Pass@8 — because autoformalization failures cluster by mathematical domain, and a mean hides exactly the pattern a practitioner needs. A number that matters on the climate side would be a comparison against a numerical storm-resolving model run on the same hardware by someone other than the emulator's authors, since the 300-times inference claim is the one that decides whether this technique gets adopted operationally.

Until then, both models are best read the way their own cards invite. MathForm-8B is a strong autoformalizer with a compiler standing behind its output, and the interesting question about it is how often the compiled statement says the right thing rather than merely a true one. HiRO-ACEv2 is a research emulator that has just added wind and pressure to what was previously a precipitation-only output, and the interesting question is whether the added fields are physically consistent with the precipitation they ship beside — something no benchmark table would have told you even if one existed.

Comparisons like this one are also where the failure modes show up: the documented AI incidents and agent failures behind models in this class are collected in the OrcaRouter AI Incident Archive , where each entry carries its own sources, severity, confidence grade and disputed flag.