Laya vs Nimble: Contrastive Data Versus a Faster Encoder

Laya and Nimble are the two open-weight decision models whose authors did the most to explain how they were built, and their explanations disagree about where the difficulty lives. Convai Innovations released Laya on September 18, 2026 under Apache 2.0 — a 421M ModernBERT-large English checkpoint, a 322M mmBERT-base multilingual checkpoint covering 100+ languages, a sub-millisecond router between them, and a published 32.8ms p50 per decision on a Tesla T4. Bespoke Labs built Nimble as a LoRA fine-tune on Qwen3.5-9B, Apache 2.0, and describes it as "the open-source Jev" — inspired by TypeSafe AI's Jev but using neither its weights, its architecture, nor a distillation of its outputs. Convai's answer to the accuracy problem is speed and a fine-tunable base. Bespoke's answer is the training data. That difference is worth more attention than the three-point accuracy gap that shows up in the headline tables.

The claim each project is actually making

Laya's claim is architectural. It is non-autoregressive in the strict sense — a bidirectional encoder, no decoding loop, no JSON to parse, no space in which to emit text outside the declared type. That structural guarantee is the same one Jev offers, arrived at from a different direction, and it is genuinely valuable for anyone who has written retry logic around a model that occasionally forgets to close a brace. The cost is that a 421M encoder with a 512-token window and no generative pretraining behind it does not know very much. Convai says so on the model card: "Laya is a fast base to specialise, not a zero-shot decision engine."

Nimble's claim is about the data. Bespoke's method is contrastive curation, adapted from the team's earlier Bespoke-MiniCheck work: write two nearly identical examples that differ in one "focus fact," such that the correct label flips. The pipeline has four stated steps — check that the decision rules could genuinely lead to different answers, build the pair by changing at most eight words, verify both examples with separate model calls plus a remove-each-sentence check, and generate labels in code while keeping only the pairs whose labels actually differ. The goal is not more examples but sharper ones: force the model to learn which evidence should change the decision. That is a different theory of why small decision models fail, and it is a more interesting one than another accuracy point.

What the published numbers actually support

Nimble reports 90.12% agreement with reference labels on 324 held-out samples, against 93.21% for Jev, 66.36% for the untuned Qwen3.5-9B base, and 84.88% for an untuned 27B Qwen3.8. It reports roughly 106ms median on an H100 and 444ms median on an M5 Pro with 64GB. Those are Bespoke's own harness figures. The 324-sample evaluation set is the part to weigh carefully, and the project says why itself: the samples span six of the ten training source families, they were generated and validated by models without human review, and generalization to unseen categories is therefore unproven. When a model is trained with a data-curation method and then evaluated on a held-out split of that same curated distribution, the accuracy number is a statement about the method's internal consistency, not about your traffic.

Laya's numbers come from the other end. On TypeSafe's typed-decisions benchmark it scores 0.362 zero-shot — random is 0.318, the majority-class baseline is 0.461 — and 0.766 when fine-tuned on the benchmark's own training split. On Banking77, 77 labels, it scores 0.425 against Jev's 0.870, which is the sharpest illustration of its many-option ceiling. On AG News with four labels it scores 0.950 against Jev's 0.910; on DAIR Emotion with six labels, 0.595 against 0.480. Its calibration error ships at 0.466 and falls to 0.081 after per-question-type temperature refitting. One third-party test of 100 Mars-base urgency messages had Jev at 100/100 and Laya at 53/100. And in an independent agent-tool-call evaluation Laya achieved 100% refusal recall on dangerous calls but only by flagging everything, which is a precision failure dressed as a safety win.

Set the two sets of numbers next to each other and the honest reading is that they were measured on different things. Nimble's 90.12% is agreement with its own curated reference labels. Laya's 0.362 is a benchmark it did not build. Neither number travels.

Calibration: the one place both projects are unusually candid

This is where the two projects are closest in spirit and furthest apart in result.

Bespoke's README warns explicitly that Nimble's probabilities are softmax-normalized logits over the supplied candidates, not calibrated correctness rates — a 0.9 does not mean 90% correct. It recommends adding a "none of the above" option, because if the right answer is not among the candidates one of them still wins. On a newer 3,880-record evaluation spanning 13 subsets, Jev had lower reported calibration error in 11 of 13 subsets and a lower Brier score in 10 of 13. So similar label agreement does not imply similar probability quality, and Bespoke says so.

Convai's disclosure is the mirror image: the 0.466 expected calibration error is on the card, next to the 0.081 that per-question-type temperature refitting produces. Neither project ships a model whose confidence you can act on without work. Both tell you that in writing. If you are building a confidence threshold — auto-release above 0.95, escalate below 0.7 — the documentation from both authors is telling you the same thing: fit the threshold on your own labelled data first.

The comparison, dimension by dimension

• Backbone — Laya: ModernBERT-large 421M encoder, bidirectional, no generative pretraining. Nimble: Qwen3.5-9B with a LoRA fine-tune, generative base retained.

• Languages — Laya: 100+ through the 322M multilingual checkpoint. Nimble: English.

• Prompt budget — Laya: 512 tokens English, 1,024 multilingual. Nimble: 2,048 tokens maximum per prompt, flat schemas only, enums capped at 26 options per field.

• Speed — Laya: 32.8ms p50 on a T4, 7.2ms per question batched at 10. Nimble: roughly 106ms median on an H100, 444ms on an M5 Pro 64GB.

• Hardware — Laya: CPU, CUDA and Apple MPS; under a gigabyte resident on the MLX port. Nimble: BF16 CUDA GPU for the stated figures, with MLX and CUDA paths supported.

• Reported accuracy — Laya: 0.362 zero-shot, 0.766 fine-tuned, 0.425 on Banking77. Nimble: 90.12% on 324 curated held-out samples against Jev's 93.21%.

• Method — Laya: architecture first, fine-tune per deployment. Nimble: contrastive data curation, published as a recipe.

• Licence — Apache 2.0 for both.

Two constraints in that list deserve to be read twice before you commit. Nimble's 26-options-per-enum cap and 2,048-token prompt ceiling are hard schema limits, not performance degradation — if your taxonomy has forty labels or your prompt carries a long document, Nimble is the wrong tool regardless of its accuracy. And Nimble scores each field independently, so any cross-field consistency rule — "if A is true then B must be false" — has to live in your code, not in the model.

Why the data recipe is the more portable artefact

Here is the argument for Nimble that the accuracy tables miss. Bespoke published the curation pipeline, not just the weights. If your decision problem has a fixed taxonomy and you can write the rules down, the contrastive method is something you can run on your own data with your own focus facts, on top of whatever base model you prefer. The 9B LoRA is a demonstration of the method as much as a product.

Laya's portability is different and complementary. Because it is small, because it runs on CPU, and because the ONNX and MLX ports exist, Laya is the model you can put inside a process that has no GPU and no network. In a third-party browser-agent benchmark Laya completed 0 of 50 tasks and declared completion prematurely in 33 attempts — but the benchmark authors note it was trained for judgment work like support tickets, invoices and agent-trace review, not navigation. Read that result as a scope statement rather than a verdict, and it is consistent with both projects' framing: these are components for a specific class of decision, not general agents.

Neither Laya nor Nimble is a hosted model on OrcaRouter. Both are weights you run yourself, and nothing in this piece should be read as a claim that we serve them. The reason the routing product comes up at all is the same reason it comes up in any decision-model architecture: the decision model answers one call, and the application around it still needs prose. A pipeline that uses Nimble to score whether a draft is compliant and Laya to route the exception is still going to need a generative model to write the draft. Keeping that generative half on one OpenAI-compatible endpoint — 200+ models, provider list price passed through with 0% markup so a vendor price cut is live the same day, automatic failover across providers — means the decision layer can be swapped without touching the generative layer's contract.

The verdict, and the experiment that would change it

Choose Nimble if your taxonomy is fixed and narrow, your inputs are English and short, you have a GPU, and you want the data recipe as much as the model. Choose Laya if you need multilingual input, a sub-40ms budget, or a process with no GPU at all — and price the fine-tuning in from the start, because the zero-shot checkpoint is below the trivial baseline and the model card says so.

The experiment that would settle it is a paired evaluation on real traffic: same inputs, same option sets, same confidence thresholds, evaluated against human labels, with a reliability diagram for each. Nimble's own documentation warns that its probabilities are not correctness rates and Laya's card reports a calibration error of 0.466 before refitting, so that diagram is the artefact that would actually tell you which model to put in front of production. Neither project has published one, and neither has published the other's. Build it on your own data before you set a threshold.