Laya on Apple Silicon: What the MLX Port Buys You, and What It Doesn't

Laya is a decision model that never writes a sentence. Convai Innovations put its weights on Hugging Face on 2026-09-18, and the next day a developer named mizorewww published Laya-MLX — an independent port that runs all three Laya checkpoints natively on Apple Silicon through MLX, with no PyTorch, no Transformers runtime and no cloud call. That port reports 13.42 ms median for one short English question on the 421M checkpoint, 7.39 ms on the 322M multilingual one, and zero output tokens, on an M3 Max. Meanwhile Kev, the other open family chasing the same typed-decision idea, is built on Qwen3.5-4B-Base and needed a whole second backend before it was usable on a Mac, because PyTorch has no kernels for its DeltaNet layers on Apple's GPU. Two projects, same week, same goal, and only one of them ported cleanly. That difference is the story, and it is a runtime story rather than a model story.

The reason this is worth an article today is not that Laya is new. It is that until 2026-09-19 there was no way to run a typed-decision model on a Mac without dragging a PyTorch stack along with it, and the question a reader actually has — can I run this on my laptop, and what do I give up — finally has a measurable answer. So this piece is about the serving path, the numbers behind it, and the places where the numbers stop meaning what they look like.

First, what Laya is not

Laya is not an LLM. It is non-autoregressive: one bidirectional forward pass over the state plus your questions, and out come typed answers. There is no token-by-token decoding, no chain of thought, no generated JSON to parse, and no output tokens to bill. The three answer primitives are choice (pick one of N named options), score (an ordinal rubric level) and noul (a calibrated probability that something is true).

That matters for how you read every number in this article. When the port reports 13.42 ms, it is not reporting 13.42 ms to produce a few hundred tokens the way a generation benchmark would. It is reporting the whole operation. Comparing a decision model's latency to an LLM's tokens-per-second is comparing two different jobs, and any article that does it — including the viral "50x faster than Jev" post that circulated after launch — is making a claim the underlying work does not support.

What the port actually measured

These figures are the port author's own, taken on a stated machine, and they should be read with the machine and the method attached. Laya-MLX measured them on an M3 Max with 40 GPU cores and 128 GiB of unified memory, at FP16, with model loading excluded.

• One short question, P50 — 13.42 ms on the 421M English checkpoint, 7.39 ms on the 322M multilingual checkpoint.

• One short question, P95 — 13.92 ms and 7.79 ms respectively.

• 50-question throughput — 146.8 questions per second and 395.0 questions per second.

• Peak MLX allocation — 943.6 MiB and 687.6 MiB.

The timing boundary is the part worth reading twice. It includes prompt preparation, tokenization, tensor construction, synchronized inference, calibration and result formatting. It excludes model loading. The 50-question throughput run used batch_size=64, while the API defaults to 16, so that pair of numbers describes a deliberately batched workload rather than what a single interactive call costs. Different input lengths, different question counts and different runtime conditions all move the result. Those caveats are the difference between a number and a benchmark, and the port states them itself.

The memory floor is the figure most readers will act on, and it is the least ambiguous one in the set: under a gigabyte of peak MLX allocation for a single short question, on both checkpoints. That is not a claim about your Mac's total footprint — the OS, your terminal and the Python process all sit alongside it — but it is a genuine floor, and it is roughly three orders of magnitude below what running a mid-size generative model locally demands.

The fidelity check is the more interesting result

A fast port that answers differently from the model it ports is worthless, and this is where the project did the work that matters. All three checkpoints matched the upstream selected answer on 63 of 63 validation questions in both FP32 and FP16 — 378 of 378 comparisons. Each configuration also ran 100 repeated, deterministic calls with no measured active-memory growth, and all 36 published weight files passed strict remote checksum verification.

Read the scope honestly: that measures fidelity on those fixtures, not accuracy on every possible question. It tells you the port is faithful to Laya. It tells you nothing about whether Laya is right.

Independent, community-maintained, and still not on the list

The port says this about itself, twice: it is an independent MLX port, not an official Convai Innovations release. RLCD training and fine-tuning stay upstream. Weights are credited to Convai Innovations. Apache-2.0 on both sides.

How upstream treats it is more revealing than any disclaimer. The Laya README carries a Community Tools list, and as of 2026-09-23 it holds four entries: omp-laya-judge, laya-adk-toolkit, laya-Ascend for Huawei Ascend NPUs, and laya-apple — an Apple Silicon runtime using the MLX GPU and the Neural Engine. That fourth entry arrived through pull request #260, merged on 2026-09-23. The port this article is about is not among the four. Upstream's list now points Apple Silicon readers at a different community project than the one that shipped first and has the benchmarks.

Upstream's own issue tracker says the rest. Issue #50, "Apple silicon ports," opened 2026-09-21, is still open; the maintainer replied the same day that Apple Silicon support is being tracked and that community ports like Laya-MLX are exploring native Metal inference, and then replied again on 2026-09-23 with a line worth quoting exactly: "The MLX port remains community-maintained." The PyTorch-side fix — MPS autocast and a transformers 4.x RoPE correction — landed as pull request #273, merged on 2026-09-23, with a reviewer in that thread flagging that it still needs combining with the separate autocast refactor in #109, which remains open. And issue #52, opened 2026-09-21, reports a Laya-MLX sidecar that grew to roughly 21.7 GB of Metal memory over several hours of operation, with vmmap attributing about 21.4 GB to the graphics subsystem rather than the Python heap; it proposes bounding the allocator cache and clearing it after each inference, and it is still open.

Put those together and the practical answer is: this runtime is not upstream-blessed, it is community-maintained by the maintainer's own description, and the one memory question that matters for a long-running sidecar is being worked in public rather than fixed in a release.

Can I run it on my laptop, and what do I give up

Install is a single pip command, and the port publishes pre-converted FP16 weights so you do not convert anything yourself:

pip install laya-mlx

Then import laya_mlx as laya, agent = laya.load("aac6fef/laya-mlx"), and call agent.predict(state, questions). Requirements are Apple Silicon, Python 3.11+ and macOS 14+. The measured environment was macOS 27.2, Python 3.12.13 and MLX 0.32.2 — and the port notes that the MLX release it used shipped macOS 14, 15 and 26 wheels while the installer selected the 26 one, and that older supported macOS versions were not tested on that machine.

What you give up, dimension by dimension:

• FP16 versus FP32 — FP16 is the default and the source of every headline number above. FP32 gives closer numerical agreement with upstream, and probabilities can differ slightly across precisions even when the selected label agrees. BF16 can be requested but is not part of the published validation matrix, so treat it as untested.

• Memory floor versus headroom — under 1 GiB of peak MLX allocation for a short question is comfortable on any M-series Mac. It is not a statement about sustained server load, and issue #52 is the reason to be careful if you plan to run this as a long-lived sidecar rather than a library call.

• Multilingual versus English — the 322M multilingual checkpoint is the faster of the two and the one that covers 100+ languages, but the port deliberately carries upstream's warning forward: the English checkpoints are not substitutes for the multilingual one. Routing between them is the intended pattern, not a nicety.

• Upstream-blessed versus community-maintained — it is the second. Nothing in the upstream release notes promises this port keeps working across upstream changes.

• Speed versus calibration — a fast port does not fix a calibration bucket that ships over-confident. Upstream clamps fitted temperatures to [0.5, 5.0], and the shipped choice:11+ bucket is 0.1006, which would sharpen logits roughly tenfold and report a coin flip as near-certainty. Fitted calibration temperatures exist for a reason; fit them on your own held-out data before you branch on a probability.

Two more limits are worth carrying, both from upstream's own tracker. action.act_probability currently carries no usable signal — it reads 1.0 for almost every input, and its raw logits ran against correctness at an AUROC of 0.30 across 396 labelled decisions (issue #185). Gate on confidence instead, which reaches 0.77 on the same items. And noul questions can follow their option labels rather than the state (issue #156) — upstream's own card reports a confident "no" on clearly positive input, most strongly on the English checkpoint. The workaround it suggests is not to reach for a different model but to reshape the question: ask it as a two-option choice with neutral keys (A/B) and your yes/no wording as the option descriptions.

Why one decision model ports cleanly and the other does not

The contrast here is architectural, and it is the most useful thing in this article for anyone choosing between the two families.

Laya's backbone is ModernBERT-large, a bidirectional encoder built entirely from attention. Attention is what Apple's GPU stack is best at, and it is what MLX has spent its effort on. So the port is a reimplementation of layers that already had fast paths: the encoder, the decision head's Transformer layers, the scoring head and the action head all run in MLX, and tokenization still goes through Hugging Face's Rust tokenizer.

Kev's backbones are Qwen3.5 bases, and Qwen3.5 mixes attention layers with Gated DeltaNet layers. DeltaNet is recurrent and ignores attention masks. That has two consequences. First, each question has to run as its own row rather than sharing one masked sequence, which the Kev project handles by computing the state once and reusing its cache per row. Second — and this is the part that bites on a Mac — there were no PyTorch kernels for those layers on Apple's GPU, so PyTorch fell back to reference code. The jaredpalmer/kev-4b model card still carries the resulting limit in plain language: a five-question request that takes 0.17 s on the Qwen3 build of Kev-4B takes 0.78 s in bf16 on an M5.

Check the current wording before quoting that, because it has moved. The Kev repository README now says the server runs the Qwen3.5 models through MLX on Apple Silicon instead, and publishes its own M5 figures for a five-question request with three options each on a roughly 270-token state: Kev-4B at 721 ms on a new state and 136 ms on a repeated state through the prefix cache, against 3,302 ms and 847 ms on the PyTorch bf16 MPS path. Kev-0.8B lands at 149 ms and 28 ms. The previous-generation Qwen3 models still run on plain PyTorch MPS and the project calls them a fine choice on a Mac.

Be careful not to turn that into a race result. These are not head-to-head measurements. Laya-MLX's 13.42 ms is one short question on an M3 Max; Kev's 721 ms is five questions with three options each on a ~270-token state on an M5. Different question counts, different option counts, different state lengths, different machines, different runtimes. What is verifiable and worth comparing is the shape of the problem, not a winner: a pure-attention encoder ports to Apple Silicon without a fight, and a hybrid linear-attention model needed a whole second backend before it was usable there.

What a decision model is actually for

Strip away the benchmarks and the honest use case is narrow, and the project says so itself: Laya is a fast base to specialise, not a zero-shot decision engine. On Convai's own typed-decisions benchmark, the two base checkpoints score 0.362 and 0.342 zero-shot against a 0.461 majority-class baseline and a 0.318 random one. They are below the line you would get by always answering the most common label. The headline 0.766 belongs to laya-typed-decisions, the checkpoint fine-tuned on that benchmark's own training split, and it should never be quoted as general capability.

Convai's published comparison against TypeSafe Jev 1.13.0 is worth reading for exactly this reason, and it is labeled carefully on their side: every Laya figure is what the router actually returns, and the Jev figures are third-party published numbers Convai never measured because it has no TypeSafe API access. In that comparison, the routed Laya scores 0.766 against Jev's 0.727 on typed-decisions, with post-temperature ECE of 0.081 against 0.246, and p50 latency of 32.8 ms against 236–276 ms on a Tesla T4 — a 7.8x difference on one question. That is the number to cite. The "50x faster than Jev" figure that spread on social media does not appear in the project's documentation or its benchmarks, and the project's own published comparison does not support it. Jev also leads where it leads: on Banking77, Jev scores 0.870 against Laya's 0.425, because Laya's options share a fixed token budget and 77 labels leave roughly three to four tokens each.

So the shape of a real deployment is a decision head that is cheap, local, and narrow — routing a ticket, scoring urgency, answering a yes/no gate — with something generative behind it for the part that needs to write. The decision model makes the typed call in milliseconds and escalates. The generative half is a different model on a different runtime, and that is where a router earns its place: 200+ models behind one key at provider list price with no markup, so a vendor price change is live the same day, and automatic failover when a provider degrades mid-run. OrcaRouter does not serve Laya, and it does not serve Kev or Jev — the Qwen3.5 family is on our model list, and the decision models themselves are not. What we cover is the generative half of that stack, which is the half you call on every request that the decision head escalates.

There is one more reason to keep the two halves separate rather than reaching for one model to do both. A local decision head that costs zero output tokens and never touches a network is a different kind of dependency from an API call: it keeps working when the network does not, and its cost does not scale with how much text it reads. That is the property worth paying for. Everything else in this article is about how much you pay for it in fidelity, memory and maintenance.

Who should run it, and who should wait

Run Laya-MLX if you are on an M-series Mac, your decisions are constrained — a choice over named options, a rubric score, a yes/no gate — and you either have labels to fine-tune on or you are prepared to fit calibration temperatures yourself. The install is one command, the memory floor is under a gigabyte, and the fidelity work has been done and published.

Wait if you need upstream support guarantees, if you are running a long-lived sidecar and want the memory-growth question settled in a release rather than an open issue, or if your questions are open-ended. A non-autoregressive encoder answering "what should I do next" is not a smaller version of an LLM doing the same thing. It is a different instrument, and it only reads well when the question is already shaped for it.