Laya Explained: A Decision Model That Answers Without Writing a Single Token

The most interesting thing about Laya is not its speed. It is that every option you offer it gets scored at its own [MASK] token, and the probabilities are then softmaxed over that one question's options. Convai Innovations published Laya's weights on Hugging Face on September 18, 2026, under Apache 2.0 — three checkpoints, one repository, 421M parameters for the English model. There are no output tokens. There is no decoding loop, no JSON to parse, no brace to forget to close. You hand it a state and a set of typed questions, and one forward pass later you get a choice over named options, an ordinal score with an expected level, or a probability that a statement is true. That design has a consequence people miss: because the answer space is assembled per request rather than baked into a vocabulary head, a schema you invent this afternoon needs no retraining. It also has a limit, and the project states it plainly on its own model card: the base checkpoints score 0.362 on the typed-decisions benchmark, against 0.318 for random guessing and 0.461 for always answering the majority class. Convai's own sentence is the one to keep in your head — "Laya is a fast base to specialise, not a zero-shot decision engine." The obvious point of comparison is TypeSafe AI's Jev, a hosted System One model with no published weights, no published parameter count and no published base model. Laya is the open-weights answer to it. Whether that answer is useful to you depends almost entirely on which half of the pipeline you are trying to replace.

What Laya actually is, and what it is not

Start with the negative, because it is where most write-ups go wrong. Laya is not an LLM. It is non-autoregressive: a single forward pass produces the answer, and the model never emits text. Comparing its latency to a chat model's tokens-per-second is comparing two different operations — one classifies, the other generates. If you need a paragraph, a summary, a plan, or a chain of reasoning, Laya cannot give you one and is not trying to.

What it is: a bidirectional encoder with a decision head bolted on top. The English checkpoint is ModernBERT-large — 395M parameters, fully fine-tuned — plus a head trained from scratch made of two transformer layers, an option-marker scorer, and an act/escalate head, for 421M total. The multilingual checkpoint swaps the backbone for mmBERT-base, 22 layers and a 256k vocabulary, at 322M total. Three checkpoints ship in the one repository, and only the one you request is downloaded:

• convaiinnovations/laya — ModernBERT-large, 421M parameters, 512-token context, English, roughly 808 MB on disk.

• convaiinnovations/laya-multilingual — mmBERT-base, 322M parameters, 1,024-token context (the encoder supports up to 8,192 with RoPE), 100+ languages, roughly 2.2x faster, roughly 647 MB.

• convaiinnovations/laya-typed-decisions — ModernBERT-large, 421M parameters, 1,024-token context, and the only one of the three that carries the 0.766 figure you will see quoted everywhere.

A Router sits in front and picks the checkpoint per request by detecting script and language in under half a millisecond, in pure Python, before any forward pass happens. That is not a convenience feature. It is a correctness feature, and the project's own evidence shows why: the English checkpoint scores 0.000 accuracy on Khmer while reporting 0.952 confidence. A model that stays confident while being completely wrong is exactly the case where confidence gating cannot save you, so the routing decision has to be made before the model sees the input. Across a 51-language sweep, the router made 45 of 51 languages usable — defined as beating three times random — against 23 of 51 for the English checkpoint alone.

The design fact worth understanding: one [MASK] token per option

If you take one thing from this article, take this. In a normal classification head, the label set is fixed at training time: the final layer has one output per class, and adding a class means retraining. Laya does not do that. It renders each option as text with a marker, and the option-marker scorer reads a score off that option's own [MASK] position. Then it softmaxes across the options belonging to that question.

The answer space is therefore defined at request time. You write the options, the model scores them. A new schema needs no retraining and no fine-tuning, because there is nothing in the weights that encodes "billing" or "technical" as a class — only the machinery to compare one rendered option against another in the context of the state.

Two budgets govern how well that works, and they are shared. Each sequence splits into an option-prompt budget (head_max_len, 192 tokens on the English checkpoint and 256 on the other two) and a document budget (whatever remains of max_len). Every question in a call is answered in that same single forward pass, so a call with six questions is not six model invocations. But options share the option budget, which is why a 77-option question like Banking77 allocates roughly three to four tokens per label and accuracy falls off a cliff — 0.425 against Jev's published 0.870. The fix is documented rather than hidden: raise head_max_len and max_len, or split a large option set into a two-step coarse-to-fine choice.

The three primitives

Everything Laya does is one of three question types, and each returns a different shape:

• choice — a probability per named option, plus the top label and a confidence. This is the routing and intent-classification primitive.

• score — a distribution over an ordered rubric plus an expected level. This is the ordinal primitive: urgency, frustration, severity.

• noul — a calibrated probability that a statement is true, from 0.0 to 1.0. Phishing, churn risk, prompt injection.

The types are strict in a way that matters operationally. A choice question cannot return an option you did not supply, because the only options it can score are the ones you rendered. That removes a whole class of production failure — the invented enum value, the truncated JSON, the retry loop around a parser. It does not remove semantic error. A model that returns billing: 0.94 for a ticket that should have gone to technical support is wrong, and it is wrong confidently. Typed output guarantees the shape of the answer, never its correctness.

RLCD, or why the probabilities are supposed to mean something

Most classifiers are trained to be right. Laya is trained to be honest about how right it is, and the training recipe is where that comes from.

The method is called RLCD — Reinforcement Learning for Calibrated Decisions. The policy emits a distribution rather than an argmax; exploration adds zero-mean Gaussian noise to the logits; and the reward is a strictly proper scoring rule — log plus spherical, with a ranked probability score added for ordinal questions. That word "proper" is doing the work. A strictly proper scoring rule is maximised in expectation only by reporting your true beliefs, so hedging or overclaiming loses reward by construction rather than by instruction. Updates are REINFORCE with a group-mean baseline, GRPO-style, and multi-turn conversations use TD(λ=1.0) over prefix slices.

The practical consequence is that a confidence threshold is a meaningful thing to build application logic on — a claim you cannot make about a softmax off a cross-entropy-trained classifier. It is also a claim with a caveat the project is upfront about: the shipped checkpoints are overconfident, and you are expected to refit a temperature on your own data before trusting the numbers. Refitting one temperature per question type and option count moved mean ECE from 0.466 to 0.081 on the English checkpoint and 0.314 to 0.106 on the multilingual one. The project's suggested starting threshold for auto-approve-versus-human-review is around 0.85.

What it costs to run

The latency figures are the project's own, measured on a Tesla T4, with every checkpoint answering byte-identical questions in the same run:

• One question — 39.5 ms on laya, 32.8 ms on laya-multilingual.

• Five questions — 84.5 ms and 40.1 ms.

• Ten questions batched — 158.6 ms (15.9 ms per question) and 72.3 ms (7.2 ms per question).

• Fifty questions — 771 ms and 337 ms, or 6.8 ms per question on the multilingual checkpoint.

• Batched throughput on a single T4 — 103 to 332 questions per second.

If you have seen a "50x faster than Jev" claim circulating, it is not the project's number and the project's own benchmark does not support it. Convai's published comparison is 7.8x on p50 latency for one question: 32.8 ms against 236–276 ms. That comparison is also the one to read carefully, because Laya's card labels the Jev side as third-party published figures that Convai never measured — it has no TypeSafe API access — and because it puts a local GPU forward pass against a hosted API call that includes network round-trip and queueing. The architectural part of that gap is real. The infrastructure part of it is not a property of the model.

On memory, the footprint is a few hundred megabytes per checkpoint, and the deployment table is worth knowing before you size a host. The lazy default keeps two checkpoints resident (English and multilingual, the only two the router chooses between automatically), so after each language's first load a switch costs detection only. Router(max_loaded=1) on a memory-constrained box reloads on every language switch, measured at 7.4 seconds median on CPU and 10.3 seconds on a T4. Router(preload=True) is the server configuration: nothing reloads, and per-request latency is the 32.8 ms GPU figure or 193–464 ms on CPU.

The honest half

This is where the piece earns its keep, because the surface around Laya is loud and the limitations are specific.

First, the headline number is a fine-tuned number. The 0.766 accuracy belongs to laya-typed-decisions, the checkpoint fine-tuned on that benchmark's own training split. The base checkpoints score 0.362 and 0.342 zero-shot against a 0.318 random baseline and a 0.461 majority-class baseline — below the trivial baseline, in other words. The project says so in its own limitations list rather than burying it, and the fine-tuned checkpoint clears the 0.735 teacher self-agreement ceiling, which is a genuinely strong result for a 421M encoder on four narrow workflows (invoice processing 0.804, security incidents 0.766, customer service 0.764, agent-trace observability 0.730). But it is a result about specialisation, not about the base model, and anyone quoting 0.766 as a general capability is misreading the card.

Second, the primitives are not equally good. By accuracy on the fine-tuned checkpoint: noul 0.857, choice 0.733, score 0.723. The project calls ordinal score "the weakest primitive" outright, with SST-5 at 0.372. If your decision surface is a 1-to-5 severity rating, that is the primitive you have least reason to trust out of the box.

Third, two behaviours are documented as bugs in the project's own issue tracker, and both will burn you in production if you do not read them. action.act_probability carries no usable signal yet — issue #185 — because the decision head's output is unnormalised at roughly 300x the encoder's scale, which saturates the act head so it reads 1.0 for almost every input. Its raw logits run against correctness, with an AUROC of 0.30 on 396 labelled decisions. Gate on confidence instead, which reaches an AUROC of 0.77 on the same items. Separately, noul can follow its own option labels instead of the state — issue #156 — because render_options hardcodes a noul's labels to false: / true:, and that label pair can dominate the answer, returning a confident "no" for clearly positive input. The documented workaround is to ask the same question as a two-option choice with neutral keys and your yes/no wording as the descriptions.

Fourth, a calibration detail that is easy to miss and worth stating precisely. The checkpoint ships a fitted temperature of 0.1006 for the choice:11+ bucket, and the loader clamps every temperature into [0.5, 5.0]. That clamp is doing you a favour. A temperature that sharp could take a genuinely split distribution and report it as near-certainty; the clamp means the worst case is a softer answer than the fit intended, and the loader emits a warning naming the affected bucket and telling you to treat that confidence as uncalibrated. Read the warnings on load rather than suppressing them.

Fifth, English-only on the repo root, and the failure mode outside English is not graceful — hence the router, and hence the recommendation to use laya-multilingual for anything that is not English prose.

The independent picture, where it exists, is narrower than the vendor picture and does not contradict it. An independent head-to-head — sysone-bench, 751 states across nine suites, dated 2026-09-21, run on byte-identical inputs with question hashes verified identical before comparison — has Jev ahead on triage, guardrails, moderation, banking77 and multilingual intent, and Laya ahead on AG News (0.940 vs 0.910) and MNLI (0.983 vs 0.867). Its confidence-gating result is the one I would actually plan around: gating at 0.85 confidence kept 58% of Laya's traffic at 0.878 accuracy, against 78% of Jev's at 0.917. That is the shape of the trade — Laya automates less of the traffic at a lower accuracy on the portion it keeps, and its own router run raises multilingual intent from 0.360 to 0.840.

The surface around it, which is unusually wide

For a project whose weights are days old, the integration surface is the part that surprises. All of this is in the upstream repository at NandhaKishorM/laya, which read 19,871 stars on GitHub when this was written, and it is Apache 2.0 throughout:

• laya-serve — an HTTP server that exposes the Router on the same POST /v1/systemone request and response shape as TypeSafe's hosted Jev API, so an existing TypeSafe client moves by changing its base URL. Note the security default honestly: it binds 0.0.0.0 with no authentication unless LAYA_API_KEY is set, in which case it requires a bearer token. A hardened NixOS module variant exists that runs under a DynamicUser systemd unit and passes the token via LoadCredential rather than putting it in the store.

• A full TypeScript port in laya-ts/ for Node and the browser, plus an ONNX agent path (laya.onnx_agent.ONNXAgent) for running an exported model on ONNX Runtime without PyTorch at runtime.

• An MCP server behind an optional extra, exposing laya_predict, laya_route, laya_preset and laya_status as tools.

• LangChain and LangGraph integrations — LayaRouter for conditional-edge routing with a confidence threshold and fallback, and LayaGuardrail.

• A Nix flake with nix run .#laya-serve and a services.laya-serve module, four compose files, a Docker image path with a documented quickstart, and a Kaggle notebook that runs the full RLCD fine-tuning loop on free 2xT4 GPUs in four to five hours over roughly 30k questions.

Apache 2.0 is the licence detail that decides whether you can ship this inside a product: it permits commercial use, modification and redistribution, and it does not require you to publish your changes or your fine-tuned weights. The obligation is the usual attribution and notice-preservation one, plus the explicit absence of a patent or trademark grant beyond what the licence states. For a decision layer that sits in front of customer traffic, that is a materially different proposition from an early-access hosted endpoint whose weights, architecture and training recipe are all undisclosed — which is what Jev is today, at $0.042 per million input tokens with output free and a text-only input surface.

Where this actually fits: a decision head in front, a routed LLM behind

The pattern worth internalising is not "decision model instead of LLM." It is a two-stage pipeline, and both stages exist because the other is bad at something.

Put Laya in front for the high-volume, narrow, machine-consumable judgment: route the ticket, classify the intent, score the urgency, decide whether this document is relevant to the query, check whether this draft violates a policy. Those calls have a fixed answer set, they happen thousands of times an hour, and a 33-millisecond local forward pass with zero output tokens is a better fit for them than a generative round trip. Then put a generative model behind it for the calls that genuinely need prose, synthesis, or reasoning over a long context — the drafting, the explanation, the escalation summary.

That is where OrcaRouter sits, and it is worth being precise about the boundary. We do not serve Laya; it is a 421M encoder you run yourself, and the whole point of it is that it runs where your data already is. Nor do we serve Jev — it is TypeSafe's early-access endpoint. What we cover is the generative half of that same pipeline: 200+ models behind one OpenAI-compatible key, at provider list price passed through with 0% markup, with automatic failover across providers. The practical reason that matters here is the seam between the two halves. The moment you start routing decisions to a generative model for the cases the decision head declined, you have a second integration, a second bill and a second failure mode. One key for the generation side, with failover if a provider degrades, means the decision layer's escalate path is a configuration change rather than a second vendor relationship. That is a small claim, and it is the true one.

Who should adopt it, and who should wait

Adopt Laya now if you have labelled data and a training loop, and a decision surface that is stable enough to be worth specialising. The Kaggle notebook exists precisely so that the fine-tuning step is not a research project, the base checkpoints load in about two seconds on CPU, and the licence lets you ship the result commercially without publishing your weights. The workloads that fit best are the ones the project already benchmarked: ticket triage, invoice processing, security-incident classification, guardrails and moderation, and agent-trace observability. Keep choice questions under roughly 20 options, calibrate a temperature on your own held-out data before you put a threshold in production, and gate on confidence, never on act_probability.

Wait if your decision needs to be right out of the box with no labelled data. A base checkpoint sitting below the majority-class baseline on the benchmark it was published against is not a zero-shot engine, and the honest reading of the vendor-versus-independent numbers is that a well-run hosted decision API is currently the stronger zero-shot choice. Wait as well if your option sets are large and you are not willing to tune the head budget, if your ordinal scoring needs to be trustworthy immediately, or if you need image, audio or long-document input — Laya is text-only and its context budget is 512 to 1,024 tokens by default, which is a selection of evidence rather than a whole document.

The thing that will decide this category is not the latency numbers, which are already good enough to stop being the argument. It is whether a small model that reports honest probabilities on a decision surface you defined, and that you can retrain on your own labels, beats calling a large generative model and parsing its output. Laya is a credible first serious attempt at the open-weights version of that question — and it is at most days old, which is the right way to read everything above. The base is the starting point, not the product.