Laya vs von: When the Vendor Benchmark Does Not Transfer

A commit-type classification task with six possible labels, where guessing gets you 8.3% and von scores 26.7%. A file-routing task where the same model scores 8.8%, barely above chance. A binary change-breadth task with an AUC of 0.513, which is a coin flip. Those numbers come from a third-party evaluation of von — the open-source System One model from wfzyx, a 395M ModernBERT-Large encoder released under Apache 2.0 on September 18, 2026, the same day as Convai Innovations' Laya. On von's own jabr v2 benchmark the picture is the opposite: 71.5% macro accuracy across 49 tasks and 869 cases, choice routing at 83.4%, and a ViZDoom result of 9.38 average kills at sub-18ms against TypeSafe AI's Jev at 5.62 kills and roughly 115ms. Both sets of numbers are real. The distance between them is the most useful thing anyone has published about this model category, and it applies to Laya too.

Two models, two backbones, one shared failure mode

von and Laya are near neighbours architecturally, which is why the transfer problem is the right lens for comparing them. Both are non-autoregressive encoders built on ModernBERT: von at 395M parameters, Laya's English checkpoint at 421M. Both expose the same three primitives — choice from a supplied list, score on an ordered rubric, noul as a calibrated probability of yes — and both implement the /v1/systemone wire format so a client written for TypeSafe's Jev can point at a local server instead. Both are Apache 2.0. Both return probabilities rather than text, and neither has a decoding loop to hallucinate in.

The differences are in the training story. von was pretrained on 2 trillion tokens of general web text, technical literature and code, then fine-tuned on a roughly 290,000-example balanced multi-domain corpus spanning operational and enterprise workflows, security and DevOps, safety and policy moderation, linguistics, triage, and adversarial reasoning. Post-training used Reinforcement Learning with Calibration Distribution, minimising a composite of cross-entropy and Brier score with lambda at 0.5, and temperature scaling converged at T=1.1692. Laya's English checkpoint is ModernBERT-large fine-tuned for the typed-decision shape, with a 512-token window, alongside a 322M mmBERT-base multilingual checkpoint covering 100+ languages behind a router, and a published 32.8ms p50 per decision on a Tesla T4.

The shared failure mode is that both are encoders trained to produce a readout, not reasoners. von's own README is explicit that it targets latency-sensitive pipelines where autoregressive models introduce 500–2,000ms of delay and nondeterministic schema-parsing errors. That framing tells you what it is for: fast, deterministic, statistically calibrated classification. It does not tell you it will work on your classification, and the independent benchmark is what happens when someone checks.

Reading von's own benchmark honestly

The jabr v2 results are owner-published and have not been independently audited, and the shape of the benchmark matters as much as the score. Forty-nine tasks and 869 cases is a reasonable breadth for a decision-model evaluation, and 71.5% macro accuracy is a strong number for a 395M encoder. The per-domain breakdown is where it gets informative: symptom triage at 100.0%, home services at 95.7%, city routing at 94.7%, choice routing overall at 83.4%. Those are the tasks the training corpus was built around — the README describes the fine-tuning data as operational and enterprise workflows, security and DevOps, safety and policy moderation, linguistics, triage and adversarial reasoning. A high score on symptom triage is a statement that the model learned the triage domain, not that it learned to classify.

The ViZDoom result is the one that gets quoted most, and it deserves a careful read. Nine-point-three-eight average kills in "Defend the Center," eight seeds, zero-shot from structured scene text, at sub-18ms latency, against Jev's 5.62 kills at around 115ms — a +66.9% improvement. It is a striking result and it is also a game environment driven by structured text where a fast reflexive policy is exactly the right shape. The project's framing of the System One paradigm — reflexive, parallel, deterministic, statistically calibrated — is a fair description of what that task rewards.

Then the third-party evaluation ran von on commit-type classification, file routing, feature detection and change-breadth scoring, and it lost to keyword and regex baselines on some of them. An AUC of 0.513 on the binary task is the sharpest figure: a coin flip, from a model whose calibration was tuned to T=1.1692 and whose README claims near-ideal expected calibration error. Both things can be true. A model can be well calibrated on the distribution it was trained on and useless on a distribution it has never seen — and in fact good calibration on the wrong distribution is worse than obviously bad calibration, because the confidence numbers look trustworthy.

The same test applied to Laya

Laya has had a version of this treatment, and the results are consistent with von's. On TypeSafe's typed-decisions benchmark, Laya scores 0.362 zero-shot against 0.318 for random and 0.461 for the majority-class baseline — closer to chance than to the trivial answer. Its own model card states the conclusion: "Laya is a fast base to specialise, not a zero-shot decision engine." Past roughly twenty options it degrades sharply, scoring 0.425 on Banking77 against Jev's 0.870. On 100 Mars-base urgency messages in one third-party test, Jev scored 100/100 and Laya 53/100. In a browser-agent benchmark it completed 0 of 50 tasks, declaring completion prematurely in 33 attempts, 17 of them before taking any action — though the benchmark authors note it was trained for judgment work like support tickets and invoices, not navigation, which is a scope statement rather than a verdict.

Where Laya differs from von is in what it claims for itself. Convai published the unflattering zero-shot number and the 0.466 expected calibration error on the card, next to the 0.081 that per-question-type temperature refitting produces. von's README publishes the flattering jabr v2 number and the ViZDoom win. Both projects are honest about what they did; only one of them leads with a benchmark where the model does badly. That is a sourcing observation, not an accusation — but if you are choosing between the two on published evidence, note that you are comparing a project that showed you its weak number against one whose weak number you had to find elsewhere.

The spec contrast, dimension by dimension

• Backbone — Laya: ModernBERT-large 421M English, mmBERT-base 322M multilingual. von: ModernBERT-Large 395M.

• Languages — Laya: 100+ via the multilingual checkpoint and a router. von: English, with no multilingual checkpoint published.

• Context window — Laya: 512 tokens English, 1,024 multilingual. von: not published in the same terms; the jabr v2 cases are short structured decisions.

• Latency — Laya: 32.8ms p50 on a T4, 7.2ms per question at batch 10. von: sub-15ms to sub-25ms claimed, sub-18ms in the ViZDoom run.

• Reported accuracy — Laya: 0.362 zero-shot, 0.766 fine-tuned on the benchmark's own split, 0.425 on Banking77. von: 71.5% macro across jabr v2's 49 tasks, 83.4% choice routing, and 26.7% on commit type in an independent test.

• Calibration — Laya: ECE 0.466 shipped, 0.081 after per-question-type temperature refitting. von: temperature scaled to T=1.1692 during RLCD post-training, claimed near-ideal ECE on its own distribution.

• Serving — Laya: pip install laya, plus ONNX, Go and Apple MLX community ports. von: Python and TypeScript SDKs, an HTTP server via von serve, pre-packaged presets for ticket triage, email security, moderation and security event triage.

• Hardware — Laya: CPU, CUDA and Apple MPS. von: NVIDIA CUDA, AMD ROCm, Apple Silicon MPS and multithreaded CPU.

• Licence — Apache 2.0 for both.

The part of von that is genuinely distinctive

von's composable patterns are the most under-discussed thing in its repository. Confidence gating, route dispatch, composite scoring and two-stage routing are shipped as named patterns alongside the presets — ticket triage, email security, moderation, security event triage — and they encode the architecture that a decision model actually needs in production. A single choice call is rarely the whole system; the useful shape is a cheap first-stage router that dispatches to a specialised second stage, with a confidence gate that escalates the uncertain cases. von ships that as a documented pattern rather than leaving it to you.

That maps cleanly onto what OrcaRouter does on the generative side, which is worth saying plainly because the two halves of the pattern are usually built by different teams. Neither von nor Laya is hosted on OrcaRouter — both are weights you download and run — and nothing here should be read as a claim that we serve them. What sits on our list is the other half: 200+ generative models behind one OpenAI-compatible key at provider list price passed through with 0% markup, so a vendor price cut reaches your bill the same day rather than at renewal. If von's confidence gate decides that 4% of requests need a frontier model, the routing DSL is what composes that decision into a single call instead of two contracts and two SDKs, and automatic failover is what keeps the expensive branch from being a single point of failure.

What to do with two models that both fail outside their training distribution

The practical conclusion from the von evaluation is not that von is a bad model. It is that a decision model's published accuracy is a claim about its training distribution, and the only way to know whether your problem is inside that distribution is to test it. von's own README benchmark table compares itself against GLiNER2, a fine-tuned Qwen3.5 4B, Laya, and TypeSafe's Jev on size, accuracy, latency and hosting — which is a useful table, and also a table where every accuracy entry except one comes from the author's own harness.

Between the two: take von if you want a broader set of published domains, a slightly smaller footprint, pre-built serving patterns for triage and moderation, and the ViZDoom evidence that it handles fast structured-text decisions well. Take Laya if you need multilingual input — von has no multilingual checkpoint and Laya's 322M mmBERT variant covers 100+ languages — or if a 32.8ms T4 figure and a 512-token English window fit your workload. Take neither without spending an afternoon labelling a few hundred examples from your own traffic and running both against them. Both projects' own documentation points you at that experiment, and the von third-party result is what happens when nobody runs it.

The one thing that would change this comparison is a paired evaluation on identical inputs with a reliability diagram for each model. It does not exist. Until it does, the honest ranking is by evidence quality rather than by score: Laya has published its worst number, von has published its best, and the independent test of von is the closest thing either has to an outside check.