LFM3-1.7B vs MiniCPM5-2B: Testing Liquid's 23-to-4 Scoreboard

On 27 October 2026 Liquid AI published a 27-row benchmark table pitting LFM3-1.7B against MiniCPM5-2B. Liquid's model wins 23 of those rows. MiniCPM5 wins four. The count circulated quickly, which is a problem, because a win count is the least informative thing you can extract from a table where one vendor chose the benchmarks, wrote the harness and ran both models. This is a test of what that 23–4 actually tells you — and it starts with an admission: the comparison Liquid is quietly answering was published on this blog a day earlier, and it declined to make this exact argument. That piece compared the two base checkpoints and concluded neither had usable evaluations. Here is the version with the post-trained weights on both sides.

MiniCPM5-2B is OpenBMB's 2.52B dense reasoning model, shipped in early September 2026 under Apache 2.0. LFM3-1.7B is Liquid AI's 1.73B text and agent model, released 27 October 2026 under the LFM Open License v1.0. They are the same weight class, they compete for the same on-device slot, and they were built for visibly different jobs.

Two vendors, two harnesses, one table

The 23–4 count only exists because Liquid re-ran its competitor. That is a legitimate and unusually rigorous thing for a launch to do — most vendors compare against a public leaderboard number and leave the scaffolding unstated. But re-running a competitor on your own harness does not produce a neutral result, and the give-away is in the rows where the same benchmark appears on both vendors' cards.

Take BFCL V4. Liquid measures MiniCPM5-2B at 62.65. OpenBMB's own card reports 66.6. Take Humanity's Last Exam: Liquid gets 8.20 for MiniCPM5, OpenBMB claims 8.9. Those gaps are small — within four points on a 0–100 scale — and they suggest the harness difference is modest on standardised benchmarks. Now take SWE-bench Verified, where OpenBMB reports 46.4 for MiniCPM5-2B and Liquid reports 43.40 for LFM3-1.7B. Put those two numbers side by side and MiniCPM5 wins. Put Liquid's own MiniCPM5 row next to Liquid's own LFM3 row and LFM3 wins. Same benchmark, same two models, opposite answer, because the numbers came from different harnesses.

This is the whole problem in one example. Neither vendor published harness configuration, retry policy or prompt scaffolding. Until someone runs both models through one harness they do not control, every cross-vendor row in this comparison is a coin flip dressed as a measurement.

Where the gap is too large to be harness noise

Some rows cannot be explained away. Agentic web navigation is the clearest:

• WebArena-Verified — LFM3-1.7B 38.42 vs MiniCPM5-2B 9.11 on Liquid's harness.

• AutomationBench — 41.29 vs 4.71.

• RULER 128K — 65.56 vs 37.96.

• BFCL V4 — 71.39 vs 62.65.

A four-point spread is scaffolding. A twenty-nine-point spread on WebArena-Verified, or a thirty-seven-point spread on AutomationBench, is not. Those are differences in whether the model can hold a multi-step plan together at all, and they line up with what each vendor says about itself: Liquid built LFM3 around agentic mid-training with tool-call traces, and OpenBMB's own documentation flags MiniCPM5 as weak on long multi-step tasks and prone to repetition. Two independent descriptions of the same weakness, arriving from opposite directions, is the strongest evidence in this whole comparison.

The long-context row deserves separate attention because it is the one people will misread. RULER 128K at 37.96 for MiniCPM5 is a retrieval score at 128K, not a statement that the model caps at 128K — MiniCPM5-2B officially supports 131,072 tokens. It means that at that length, MiniCPM5 finds less of what you ask it to find. LFM3 scores 65.56 at the same length. If your workload is "search a long document and answer", that gap is the entire decision.

The four rows MiniCPM5 wins — and why they matter

Liquid's table hands MiniCPM5-2B four victories: τ³-bench at 69.44 against 68.01, GPQA Diamond at 65.15 against 63.37, HLE at 8.20 against 7.17, and AA-Omniscience at 9.13 against 7.07. Three of the four are knowledge benchmarks. The margins are one to two points, which is inside the noise floor of most evaluation suites.

But the pattern is consistent with MiniCPM5's design. OpenBMB trained it as a reasoning model with an explicit thinking mode, on a chain that ran deep-thinking SFT over 400 billion tokens and then reinforcement learning with specialist teachers before on-policy distillation into a single dense checkpoint. That is a recipe aimed at getting more out of stored knowledge per parameter. LFM3's training is aimed at tool calls and instruction adherence. GPQA Diamond going to MiniCPM5 by 1.8 points is not a surprise; it is the two training philosophies showing up where you would expect them.

The far more interesting comparison is the one Liquid's table cannot make. MiniCPM5-2B has an Artificial Analysis Intelligence Index of 12. LFM3-1.7B, as of 28 October 2026, has no Artificial Analysis entry at all — no index, no independently measured speed, no neutral latency figure. So the model that loses 23 of 27 rows is the one with an independent datapoint, and the model that wins them is the one asking to be taken on trust. That asymmetry should weigh more than the win count does.

The specs that actually compare cleanly

Where neither vendor is running the other's benchmark, the numbers are boring and reliable:

• Parameters — LFM3-1.7B 1.73B dense vs MiniCPM5-2B 2.52B total, 1.98B non-embedding.

• Layers — 24 hybrid (13 convolution, 5 sliding-window, 6 global) vs 42 standard transformer layers with grouped-query attention at 16 query and 2 KV heads.

• Context — 256K vs 131,072.

• KV cache at 128K — 1.6 GB vs 5.6 GB, on Liquid's measurement of both.

• Footprint at 4-bit — roughly 1 GB vs roughly 1.1 GB in hands-on reports.

• Vocabulary — 128K vs 130,560.

• Licence — LFM Open License v1.0, free below $10M revenue vs Apache 2.0, no threshold.

The KV-cache row is the one that decides hardware. A 3.5× difference at the same context length is the difference between a long-context agent that fits on a phone and one that does not. It also explains why LFM3 can offer 256K where MiniCPM5 offers 131K: the tri-hybrid layer mix is cheaper per token of context, which is the entire point of building it.

The licence difference matters more than the memory difference for some teams. Apache 2.0 has no revenue threshold, no commercial agreement to negotiate and no ambiguity about redistribution. The LFM Open License is free below $10 million in annual revenue and requires a conversation above it. A startup under that line gets both for nothing. A company over it gets one for nothing and one after a phone call, and that is a real procurement cost regardless of which model scores higher.

Ecosystem support is where MiniCPM5 still leads

Both families shipped broad runtime support on day one, so this gap is narrower than it was a year ago. LFM3 lists vLLM, SGLang, Transformers, llama.cpp, MLX, ONNX, Ollama and LM Studio, with BF16, FP8, NVFP4, GGUF, MLX and ONNX builds. MiniCPM5-2B lists the same runtimes plus GPTQ and DSpark, a speculative-decoding draft model that pairs with the main checkpoint.

The practical difference is maturity rather than availability. MiniCPM5-2B has been downloadable since early September, which is seven weeks of community bug reports, quantisation experiments and framework patches that LFM3 has not had. A model released yesterday will have rough edges in its GGUF conversion and its Ollama packaging. That is not a criticism of Liquid; it is what day one looks like. If you need the model in production this week, seven weeks of other people hitting the problems first is worth something.

How to decide, and where a router earns its place

Neither model is in the OrcaRouter catalogue — both are open weights you run yourself, and we would rather say that plainly than imply otherwise. The router matters at the boundary. The honest way to run this evaluation is to put both models behind one endpoint and send real traffic at them, and the honest way to run either in production is not to let a 2B model be the only thing that can answer.

That is what automatic failover is for. Route a request to LFM3-1.7B, let it handle the tool call or the extraction it was trained for, and fall through to a hosted model when the task turns out to need more steps than a 2B model can hold. Because OrcaRouter passes provider list price through at 0% markup, the expensive path only costs what the provider charges when it is actually used — the saving from routing to a small local model is not eaten by the router. The routing DSL lets you express that as a rule instead of a rewrite: specialist first, generalist on failure or on a confidence threshold.

Model fusion is the underused option here. When two 2B models disagree about a classification, having both answer and reconciling is often cheaper than escalating to a 27B model, and the disagreement rate is itself a useful signal about which inputs are hard. With an unverified model on one side of your pipeline, that kind of cross-check is cheaper than trusting a vendor table.

What would settle it

Thirty days from now, someone should publish LFM3-1.7B and MiniCPM5-2B through one harness with the configuration committed alongside the results. If LFM3 holds anywhere near 38 on WebArena-Verified and 65 on RULER 128K, the 23–4 count will have been approximately right for the wrong reasons and the agentic gap is real. If it drops toward MiniCPM5's numbers, then the launch table was measuring scaffolding.

Until then, the defensible position is narrower than either vendor's marketing. If your workload is multi-step, tool-calling or long-document, LFM3-1.7B is the better bet and the memory advantage makes it the only one of the two that fits on constrained hardware with headroom. If your workload is knowledge retrieval, single-turn reasoning, or needs an unambiguous licence and a model with an independent score attached, MiniCPM5-2B is the safer pick and the four rows it wins are pointed in exactly that direction. The 23–4 count is not wrong so much as unearned — and the model count in a launch table is not the same thing as a result.