SingVL_S1_4B vs LFM2.5-VL-3B: Probabilities Against Paragraphs

Asking whether SingVL_S1_4B beats LFM2.5-VL-3B is a malformed question, and the fastest way to waste a week is to answer it anyway. One of these models writes paragraphs about images; the other returns a probability for each option you hand it and never emits a word of its own. That difference is not a quality gap to be settled on a leaderboard. It decides which of your pipelines can use which model, before any benchmark is consulted. So the honest comparison runs in the other direction: start from what your code needs to receive, and see which model can produce it.

Two different return types

Liquid AI's LFM2.5-VL-3B is a vision-language model in the ordinary sense. Give it an image and a question and it generates an answer token by token, which means it can describe a scene, follow a multi-turn conversation, call a function, transcribe a page of text with layout annotations, or return bounding boxes for objects you name. It was released on August 12, 2026, built on the LFM2.5-2.6B language backbone with a SigLIP2 NaFlex vision encoder, and it is explicitly pitched at on-device deployment — 228 tokens per second on an Apple M5 Max in under 3.3 GB of memory, with GGUF, ONNX and MLX builds published alongside the native checkpoint.

inclusionAI's SingVL_S1_4B, uploaded quietly on November 11, 2026, does one thing and does it in a single forward pass. You give it a state — text, JSON, up to four images — and a dictionary of named questions, each declaring a type (choice, yes/no under the label "noul", or an ordered 2-to-10-level score) and its candidate answers. It returns a probability per candidate. There is no sampling step, no temperature to set, no output to parse. The model card itself is written in that vocabulary throughout, down to the API examples.

If your endpoint needs to return "the invoice is from Acme, dated March, total 1,240.00", LFM2.5-VL-3B can do it and SingVL_S1_4B cannot. If your endpoint needs {"is_invoice": 0.97, "quality": {"1": 0.02, "2": 0.11, "3": 0.87}}, the reverse holds. That is the whole decision, and everything below is about the edges of it.

The spec sheet, side by side without a table

• Parameters — SingVL_S1_4B is about 4.54B in BF16 across two shards totalling 9.1 GB; LFM2.5-VL-3B is 3.1B, with a 2.6B language backbone.

• Output — SingVL_S1_4B returns per-option probabilities from one forward pass; LFM2.5-VL-3B returns generated text, tool calls or structured layout annotation.

• Context — SingVL_S1_4B accepts 65,536 input tokens for text only, dropping to 10,000 once images are attached; LFM2.5-VL-3B advertises 32,768 tokens end to end.

• Images per request — SingVL_S1_4B caps at four, with a processor maximum of 1,806,336 pixels each; LFM2.5-VL-3B takes single and multi-image input and splits large images into 512×512 patches plus a thumbnail.

• Languages — LFM2.5-VL-3B lists fifteen, from Arabic to Vietnamese; SingVL_S1_4B's card makes no language claim at all.

• Licence — LFM2.5-VL-3B ships under the LFM Open License v1.0 with quantised derivatives already in the wild; SingVL_S1_4B ships with no licence field and no LICENSE file.

• Deployment — LFM2.5-VL-3B has native, GGUF, ONNX and MLX builds, plus a WebGPU demo and vLLM, SGLang and llama.cpp paths; SingVL_S1_4B means cloning the repository and running its own FastAPI server on port 8173.

• Ecosystem — LFM2.5-VL-3B has been on the Hub since August and has picked up community quantisations and ports; SingVL_S1_4B was one day old at the time of writing, with zero downloads.

Two of those lines are more decisive than they look. The licence line is the first: a repository with no licence grants nobody the right to redistribute or ship, and "the vendor has not said" is not a permission you can put in a compliance review. The deployment line is the second: an ONNX build that runs on a phone and a Python server that expects a CUDA device are not competing for the same slot in your architecture, whatever the parameter counts say.

Why the benchmark numbers cannot be subtracted

Both cards report scores on datasets whose names overlap, and the overlap is a trap. SingVL_S1_4B reports 82.17 on RealWorldQA, 72.11 on BLINK, 50.17 on MMMU-Pro vision and 96.31 on InfographicVQA. LFM2.5-VL-3B reports 73.1 on RealWorldQA, 61.5 on BLINK, 30.5 on MMMU Pro and 81.3 on ChartQA. Laid in a row, that looks like a rout. It is not a comparison of anything.

The first reason is the answering mode. Every SingVL_S1_4B figure is produced by scoring a set of supplied options and taking the highest-probability one, which is a fundamentally easier task than generating a free-form answer that a grader then has to match. The Decision Index measures exactly this and says so: its scores are chance-corrected, so guessing carries no credit, but the model still never has to produce the string — only rank candidates someone else wrote. A generative model answering the same question from scratch is doing more work for the same point on the chart.

The second reason is who ran it. Both sets are vendor-reported, on each vendor's own harness. Neither has been reproduced by a neutral lab. SingVL_S1_4B is additionally one day old, with no submission to the Decision Index's public reproduction kit — the card includes the instructions for making one, which indicates intent and proves nothing. LFM2.5-VL-3B has had three months of people running it, which is not the same as an audited number but does mean its figures have at least been looked at by strangers.

The third is version drift. LFM2.5-VL-3B's own card marks its OCRBench v2 figure as the English-only subset. SingVL_S1_4B's InfographicVQA row is reported inside a suite with its own normalisation. Two numbers with the same dataset name and different subsets are not the same number, and the honest move is to stop subtracting them.

The efficiency claim belongs to one side only

LFM2.5-VL-3B's headline is not accuracy, it is throughput in a memory envelope: under 3.3 GB, 228 tokens per second on an M5 Max, 116 on a Ryzen AI Max+ 395, and 20 on a Galaxy S26 Ultra. That last figure is the one that matters for product design, because it is the difference between a feature that runs on the device and a feature that needs a round trip. Nothing in the SingVL_S1_4B repository makes a comparable claim, and there is no quantised build to make one with — you get BF16 shards and a server script.

The flip side is that a single forward pass has a cost profile a generative model cannot match. Scoring four options for one question is one pass; generating a 200-token description is 200 sequential passes. If your task is genuinely a classification — is this document a receipt, does this frame contain a person, is this listing compliant — SingVL_S1_4B's design should be cheaper per decision, and it returns a calibrated number that a threshold can act on rather than a sentence a parser has to survive. That advantage is structural and does not depend on the vendor's benchmark being right. It also has not been measured by anyone outside the lab, so treat it as an expectation rather than a fact.

What it costs to find out

LFM2.5-VL-3B can be evaluated on a laptop this afternoon. The MLX build runs on Apple silicon, the GGUF build runs on CPU through llama.cpp, and there is a hosted playground if you want a look before you download anything. The switching cost is a model identifier in a config file, and the risk is bounded by the fact that the weights are licensed and the format is standard.

SingVL_S1_4B costs a GPU and an afternoon of setup. You need Python 3.12, a PyTorch build matched to your card, the repository's own requirements, and a free port. The server speaks a single route — POST /v1/systemone — with a flat JSON body, which is unusually easy to wire up for a research artefact, and the card supplies working curl examples for both text and image requests. What you cannot do is point an existing OpenAI-shaped client at it, because it is not that shape. The endpoint is designed for the Decision Index evaluation kit, and the kit is designed for it.

That asymmetry is the practical answer to "which should I try first". If you have not yet established that a probability-returning model helps your pipeline, the cheap experiment is to run the pattern against something already served. OrcaRouter carries TypeSafe Jev 1.14 at $0.042 per million input tokens under one key, which is a working decision endpoint you can benchmark your own task against in an hour, with no checkpoint to host and no licence question to escalate. SingVL_S1_4B is not available there — it has no hosted provider at all — so the comparison you can actually make today is "does typed-decision scoring beat generation on my data", not "does this particular new checkpoint win". Get the first answer before spending on the second.

Where each one belongs

• Documents you must read and quote — LFM2.5-VL-3B. It produces the transcription, with layout labels, and its OCR row is a real capability rather than a score.

• Screens, forms and GUIs — LFM2.5-VL-3B. ScreenSpot-v2 at 80.7 and the grounding improvements over the previous generation are the point of the release.

• Anything that runs on a phone or a laptop — LFM2.5-VL-3B, and there is no contest, because the alternative needs a server.

• A gate in front of an expensive call — either, and SingVL_S1_4B's shape is the better fit if you can host it, because a probability is what a gate needs.

• Multi-label triage over a fixed taxonomy — SingVL_S1_4B, in principle. Its choice and score question types map onto taxonomies directly, where a generative model has to be constrained back into one with prompts and post-processing.

• Moderation — neither without work. SingVL_S1_4B's own card shows its hateful-meme row collapsing from 71.64 to 43.28 when the skill prompt is applied, which is a warning about prompt sensitivity in exactly the domain where false negatives are expensive.

• Commercial deployment this quarter — LFM2.5-VL-3B, on licensing grounds alone.

The part that is still unmeasured

The interesting question is not which model is better, because they are not in the same competition. It is whether a 4.5-billion-parameter classifier trained on top of a vision backbone can match a general-purpose vision model on the narrow slice of tasks where you only need a label and a confidence. If it can, it changes the cost of a lot of pipelines, because a single forward pass is a different order of expense than a generated paragraph. Nobody outside inclusionAI knows yet — the repository is a day old, the licence is blank, and the benchmark is the lab's own.

Meanwhile the general-purpose model is available, licensed, quantised, documented and fast enough to run on a phone. When a team asks which of these to put in production, the answer for now is the boring one, and the interesting one is a hypothesis with a repository attached.