Ternary Bonsai 2 27B: What Fits in 5.9 GB, and What the 98.2% Doesn't Tell You

Ternary Bonsai 2 27B is a 27.36-billion-parameter multimodal language model that Prism ML announced on 17 September 2026, and the thing to understand about it is that its language weights take one of exactly three values. Its base model is Qwen3.8 27B — a 27B hybrid-attention model — and Bonsai keeps that architecture, that training, and that shape, and replaces the language model's matrix weights with a ternary representation. The shipped file is 5.93 GB. The full-precision reference is 53.81 GB. The vendor's headline claim is that it retains 98.2% of the original's benchmark average.

Start with the part most coverage will skate past: that 98.2% is Prism ML's own number, measured on Prism ML's own 20-benchmark suite, with Prism ML's own harness, and nobody outside the company has reproduced it. That is not an accusation — it is the normal state of affairs one day after a release, and it is exactly the status you should assign to it. What you can verify independently today is the file: the Hugging Face API lists Ternary-Bonsai-2-27B-PTQ1_0.gguf at 5.947 GB against the FP16 reference at 53.808 GB, which is a 9.05x reduction and matches the vendor's "roughly 9x" without needing to trust anyone. The size is a fact. The quality retention is a vendor measurement. The interesting material is in between — the category breakdown, which shows precisely where the compression is free and where it is not.

There is also a second face to this release. On 18 September, one day after the announcement, OrcaRouter published a runtime-abliterated variant of the same model — the OrcaRouter Ternary Bonsai 2 27B Uncensored — which removes a learned refusal direction at inference time and leaves the weights bit-identical. It is covered in its own section below, because the technique is the interesting part and because its limits are as instructive as its results.

It is a compressed Qwen3.8 27B, not a newly trained model

This distinction is the difference between explaining the release and repeating a press release. Prism ML did not train a 27B model from scratch and did not run a new pretraining recipe. What it did was take Qwen3.8 27B and change the numerical representation its weights are stored and computed in.

The architecture is unchanged, and it is the base model's: a hybrid-attention design that is roughly 75% linear attention and 25% full attention, with SwiGLU MLP blocks, RoPE and RMSNorm. That hybrid backbone is also why the 262K-token context is described as full-context capable rather than merely supported — mostly-linear attention is what keeps a long context affordable on a device. The model is a vision-language model: it accepts images as well as text, and the vision tower is the stock, unquantized Qwen tower, packaged separately.

What Prism ML contributed is two things. The first is the ternary representation itself plus the quantization-aware training that makes it survivable. The second is the kernels — custom low-bit kernels for that hybrid-attention stack on Apple Silicon and CUDA, which operate directly on the packed weights rather than unpacking them into FP16 and multiplying. Without the second contribution the first is a storage format with no way to use it at speed.

Prism ML's own whitepaper reports the parameter split as 24.35B in the language backbone across 64 blocks, 2.54B in the embedding and LM head, and 0.47B in the 27-block vision tower, for 27.36B total. The vision tower is the one part that is genuinely a different artifact: the GGUF release packages it as a 4-bit mmproj file of about 0.63 GB, loaded only when an image actually arrives, so text-only serving never carries it.

This is the second-generation Bonsai from the same lab; the first Bonsai 27B landed in July 2026, roughly two months earlier, and the comparison between the two generations is a reasonable question — one we take up in the head-to-head against Bonsai 27B rather than duplicating here.

What "ternary g128" means, concretely

If you have not met ternary weights before, this is the paragraph that makes everything else legible, so here it is without shorthand.

An ordinary weight in a neural network is a 16-bit floating-point number — about 65,536 distinguishable values on a useful range, each one costing 16 bits to store. A ternary weight is not a small float. It is a choice among three symbols: −1, 0, or +1. That is the entire vocabulary. Store one such symbol naively and you would spend two bits per weight, since two bits give you four states and you only need three.

On its own that would be a catastrophic loss of expressiveness, and it is why the format is never just the symbol. Each group of 128 consecutive weights shares one FP16 scale factor, and the actual weight value is the ternary symbol multiplied by that scale:

• w = ssub>g/sub> · t, where t ∈ {−1, 0, +1} and ssub>g/sub> is one shared FP16 scale for the group of 128

So the model still represents a wide range of magnitudes — it just represents them in coarse, group-wise steps instead of per-weight ones. The 0 is not a rounding artifact; it is a real third state, and having it is what lets a group of 128 weights be mostly silent when it needs to be.

The rotated basis is the part that surprises people. Before the ternary assignment happens, each weight matrix is transformed blockwise by an orthogonal rotation — a Walsh–Hadamard matrix combined with a fixed diagonal of ±1 signs, at block size 1024 — and the ternary values are chosen in that rotated space. The rotation is folded into the stored weights during preparation, so it costs no extra bits and no extra weight traffic. At inference the runtime applies the matching transform to the activations instead, and the packed model declares its rotation in its metadata, so a runtime either applies the matching transform or refuses to load the file.

Why bother? Because a Hadamard rotation spreads the energy of a weight matrix more evenly across coordinates, which makes the subsequent three-level quantization much less damaging than it would be on the raw, spiky distribution. The rotation is not decoration; it is the reason a ternary model can retain anything like the parent's quality. The cost is that the transform sits on the critical path of every projection at batch size 1, which is a real engineering problem — Prism ML fuses the sign flip into the transform's load path on Metal and parallelizes it across a full thread block on CUDA to keep it from dominating decode.

The numbers, carefully: 1.585, 1.71, 1.72, 1.76

Four bit-width figures circulate around this release, they are all correct, and they measure four different things. Conflating them is the single easiest error on this story. Here is each one and what it actually covers.

• 1.585 bits per weight — the information content of one ternary symbol, log₂3. This is a property of the format, not of any file. Nothing shipped runs at 1.585 bits/weight.

• 1.71 bits per weight — the ternary tensors only. Add the 16-bit FP16 group scale amortized over 128 weights and you get log₂3 + 16/128 ≈ 1.71. Still not a shipped figure; it is the ternary tensors in isolation.

• 1.72 bits per weight — every parameter in the language model, including the small set held above the low-bit representation. Prism ML keeps 26,238,464 parameters — 0.0976% of the language model, about 52 MB at bf16 — in higher precision, mostly the recurrent state path of the linear-attention layers plus normalization weights. Those tensors are neither rotated nor quantized, and they are what move the figure from 1.71 to 1.72. At 1.72 the idealized footprint is 5.80 GB, a reduction of about 9.3x. This is Prism ML's "True Ternary" row, and it is a target rather than a file you download.

• 1.76 bits per weight — the actual shipping GGUF. Efficient kernels need a packing format, and Prism ML's PTQ1_0 packs trits densely, landing at 1.76 bits/weight in 5.93 GB, about 9.1x. This is the file behind both the "5.9 GB" and the "9x smaller" that the announcement quotes, and it is the one the measurements above confirm.

The second packing is PQ2_0, which stores each trit in a 2-bit slot rather than densely. It costs more space for cheaper unpacking: 2.16 bits/weight in 7.25 GB, about 7.4x. Neither packing is uniformly faster — PTQ1_0 moves roughly 18% less weight data per step but pays arithmetic to unpack dense trits, so it wins on the Ada-generation cards and the L4 where memory is the binding constraint, and loses on Hopper, Blackwell and Apple silicon where batch-1 decode is limited by instruction throughput instead. Prompt processing favours PQ2_0 everywhere, because it is compute-bound.

Two housekeeping notes for anyone checking these against the sources. First, Prism ML's own documents round slightly differently — the whitepaper's storage table gives PTQ1_0 as 1.76 bits/weight at 5.93 GB, while the GGUF model card on Hugging Face gives 1.75 and 5.95 GB, and the measured file is 5.947 GB. These are the same file described at different precisions, not a disagreement about substance. Second, the announced reduction of "more than 9x" is the vendor's; measured against the actual files it is 53.808 / 5.947 = 9.05x, which is consistent.

The benchmark picture: not the average, the shape

The headline is an average of 83.9 against 85.4 for the Qwen3.8 27B FP16 baseline, which is 98.2%. The average is the least interesting part of it. The shape underneath is where the real information is, and it is not uniform.

• Instruction following — 82.66 vs 81.25. This is the one category where the compressed model beats its full-precision parent. It is not noise anyone can explain away casually; it is a category win on the vendor's own suite.

• Math — 96.57 vs 97.06, and coding — 81.58 vs 82.17. Both essentially level: half a point and six-tenths of a point on category averages. For a model at a ninth of the footprint, these are the results the whole technique is being argued on.

• Knowledge and reasoning — 83.95 vs 86.66. A 2.7-point drop, and this is where a meaningful share of the missing 1.8 points on the total average lives.

• Vision — 78.59 vs 81.64. A 3.05-point drop, the largest single category loss. Worth noting that the vision tower itself is not the compressed part; the language model reading its outputs is.

• Agentic and tool calling — 77.57 vs 79.74. The category average covers τ 2-Bench at 80.22 and BFCL v3 at 74.92.

Individual results worth knowing, because they do not all point the same direction. On Terminal-Bench 2.1 the model scores 52.8 against 69.7 for full precision — roughly three quarters — and on SWE-bench Verified it scores 60.8 against 80.6, again about three quarters. This was the first time this model family was evaluated on Terminal-Bench, and Prism ML is explicit that the long-horizon software-engineering gains it promised in the first Bonsai release are partial, not complete. Against that: τ 2-Bench rose to 80.2 from 73.6 in the previous release, BFCL v3 holds at 74.9, and AA-LCR sits at 77.0, within a point of full precision. AIME26 lands at 95.83 and LiveCodeBench at 90.07.

Where to trust it and where not to. Trust the shape on math, coding and instruction following — those are the categories where the technique is demonstrably doing what it claims, and they are measured on the same harness as the baseline. Be cautious about long-horizon agentic work: the two benchmarks that actually stress sustained tool-driven engineering, Terminal-Bench 2.1 and SWE-bench Verified, show a materially larger gap than the aggregate implies, and the vendor says so rather than hiding it. And treat the whole table as one lab's measurement on one harness until someone else runs it. That caveat is not a formality here — it is the difference between "this model retains 98.2%" and "this model's vendor measured 98.2% on a suite the vendor chose." Both are true; only one is a fact about the model.

Why this beats an IQ2_XXS build of the same base model

This deserves its own section rather than a line, because it is the entire argument for quantization-aware ternary training over post-training quantization.

The conventional way to make Qwen3.8 27B small is to quantize it after training. The whitepaper's comparison point is an IQ2_XXS GGUF build of the same base model:

• Ternary Bonsai 2 27B — 1.76 bits/weight, 5.93 GB, 20-benchmark average 83.9

• Qwen3.8 27B IQ2_XXS — 2.2 bits/weight, 7.3 GB, 20-benchmark average 75.2

The compressed-by-training model is both smaller and better. It is 1.23x smaller than the conventional low-bit build and scores 8.7 points higher. That combination is not a rounding curiosity; it is the claim that a representation chosen during training is worth substantially more than the same nominal bit budget applied afterwards.

The more instructive part is how the conventional build fails, because the failure is selective and easy to miss. IQ2_XXS does not degrade evenly. It holds up on surface knowledge — 85.79 on MMLU-Redux — while collapsing on tasks that require sustained chains of reasoning: 78.6 on AIME26, 70.05 on LiveCodeBench, 65.45 on GPQA Diamond. Bonsai 2 scores 95.83, 90.07 and 85.76 on those same three. A casual chat test would find the IQ2_XXS build perfectly serviceable and never surface the collapse; the damage lives exactly where long reasoning and code generation happen. That asymmetry is why "it felt fine when I tried it" is not evidence about a quantized model.

Prism ML compresses the same argument into a single derived figure it calls intelligence density — roughly, benchmark capability per gigabyte. On the 20-benchmark suite it reports 0.444 per GB for Bonsai 2, 0.276 for the IQ2_XXS build, and 0.051 for FP16. The metric is the vendor's own construction and its weighting is a design choice, not a law; but the ordering it produces is the same ordering the raw table produces, so it adds interpretation rather than evidence.

One more honest note on the comparison. Prism ML's GGUF model card reports a second, narrower evaluation — a 14-benchmark thinking-mode suite — on which the same retention figure appears again at 84.78 against 86.32, with IQ2_XXS at 72.59. Two different suites landing on the same 98.2% is mild corroboration that the aggregate claim is not an artifact of one benchmark selection. It is still the same lab running both, on the same harness. Our fuller breakdown of this matchup, including the packing-format question, is in the comparison against the Qwen3.8 27B GGUF builds.

What it actually takes to run

The throughput numbers, from the whitepaper's standardised tg128 measurement at batch size 1 with the vision tower excluded:

• Apple M5 Max — 46.8 tok/s decode, 765 tok/s prompt processing

• Apple M5 Pro — 27.7 tok/s decode; a separate longer-window run of the PQ2_0 pack measured 27.0 tok/s sustained, drawing 27.0 W on the GPU rail and 32.8 W across CPU and GPU

• Apple M4 Pro — 18.0 tok/s decode, with prompt processing at roughly 125 tok/s becoming the binding constraint for very long contexts

• NVIDIA RTX 5090 — 142.5 tok/s decode on the PQ2_0 pack at 0.582 mWh per token

The practical claim Prism ML makes is not a speedup ratio but an absence: the FP16 baseline at 53.8 GB does not fit on a 16 GB laptop at all, so the meaningful statement is that a 27B-class model now runs interactively on everyday hardware. On the M5 Pro the measured decode streams about 201 GB/s of weights, confirming the memory-bandwidth-dominated profile the low-bit representation is designed to exploit.

Then the edge cases, which matter more than the peak numbers.

You cannot use stock llama.cpp. The ternary hybrid-attention kernels live in Prism ML's own llama.cpp fork. Stock llama.cpp rejects the PTQ1_0 and PQ2_0 types as unknown, and — more dangerously — loads the older Q2_0 ternary format without any warning and produces garbage, because it has no Hadamard activation runtime. If you run this model on a binary that does not apply the matching rotation, you will not get an error; you will get fluent-looking nonsense. This is the single most likely way to waste an afternoon on this release.

The MLX pack has no CUDA path. The MLX release (prism-ml/Ternary-Bonsai-2-27B-mlx-2bit) targets Apple Silicon, where it has custom kernels for the hybrid stack in both the Python and Swift runtimes. Its quantized matmul has Metal and CPU kernels but no CUDA implementation, so on an NVIDIA machine that particular pack does not get GPU acceleration at all. CPU inference works, but a 27B forward pass on CPU can take minutes — which makes the Linux CPU path useful for implementation testing and reproducibility, and useless for serving.

The two packs are a genuine trade, not a ranking. If you are on an Ada-generation card or an L4, or memory is the binding constraint, PTQ1_0 is the pick at 5.93 GB. If you are on Hopper, Blackwell or a 5090, PQ2_0 buys you decode speed for 1.3 GB. If you are on Apple silicon, note that the M5 Pro figures above are measured on PQ2_0, which is also the pack the demo setup downloads by default.

A note on the MLX pack's own accounting, because it is a common source of confusion. The MLX container is an affine 2-bit format whose block stores both an FP16 scale and an FP16 bias for every group of 128 weights. Bonsai's ternary weights need only the scale — the levels come out of the scale alone — so the bias is dead weight, and the block costs 36 bytes per 128 weights instead of 34. That pushes the MLX pack's packed rate to 2.250 bits/weight, not 1.72 and not 1.76. It is a different container carrying the same ternary values, and its measured file on Hugging Face is 8.005 GiB.

The runtime-abliterated variant

On 18 September, OrcaRouter published the OrcaRouter Ternary Bonsai 2 27B Uncensored, which applies refusal-direction ablation to this model entirely at runtime. The engineering idea is worth more attention than the product, so here is the idea first.

Conventional abliteration edits weights. It finds a direction in activation space that corresponds to refusal behaviour, then orthogonalizes the weight matrices that write into the residual stream against it: W ← W − r(rᵀW). On an ordinary FP16 model that is fine — the edited matrix is still a dense floating-point matrix, so you save it and move on. On a ternary pack it is a dead end, and specifically it is a dead end for the reason this whole model exists. Orthogonalizing a ternary matrix produces a dense full-precision matrix. To store that back into the ternary pack you would have to re-quantize — and re-quantizing edited weights does not reproduce the quantization-aware training that produced the original. You would throw away exactly what was bought.

So the projection moves to inference time instead. Rather than changing W, change its output:

• y ← y − α · dot(y, r) · r, computed in float32, where y is a residual contribution and r is the normalized refusal direction

At α = 1 the component of each residual write parallel to the refusal direction is removed. At α = 0 the model is untouched. α above 1 over-projects and may degrade quality. Because α is a runtime parameter rather than a checkpoint property, the same pack can be A/B tested against itself in the same process — which is precisely what OrcaRouter's evaluations do. The original Bonsai pack stays bit-identical: zero weights modified, zero re-quantization, zero additional weight quantization error.

Two implementation details are where a naive version of this fails.

129 intervention sites, not 16. Every module that can write into the residual stream has to be wrapped, and in this hybrid architecture that is 64 mlp.down_proj blocks, 48 linear_attn.out_proj layers, 16 self_attn.o_proj layers, and model.embed_tokens — 129 in total. Wrapping only self_attn.o_proj is the obvious mistake and it catches 16 of them, leaving the other 113 writes unprojected. A self-check script measures whether the remaining component along the refusal direction is driven to roughly 1e-6 of the residual norm, and warns if it does not detect all 129 sites.

Do not re-rotate the direction. The ternary pack keeps its projections in a rotated basis on their input dimension and compensates on the activation side. The refusal projection operates on the outputs of those projections, which are already back in the normal hidden basis — so the refusal direction is an ordinary 5120-dimensional vector and applying an additional Hadamard rotation to it would project against the wrong basis entirely.

What OrcaRouter measured — our own numbers, not independent ones

These are OrcaRouter's own rule-based measurements, and they should be read as such: a rule-based opening-phrase classifier, not an LLM judge, thinking off, greedy decoding, 64-token budget, with base and ablated being the same weights in the same process at α = 0 against α = 1. They are indicative, not publication-grade, and they are not verification of anything Prism ML claimed.

On refusal, measured as the share of prompts that received a refusal:

• AdvBench (n=100) — 99.0% base, 6.0% ablated, with 56.0% answered but wrapped in a disclaimer

• JailbreakBench (n=100) — 96.0% base, 4.0% ablated, 52.0% caveated

• StrongREJECT (n=150) — 99.3% base, 3.3% ablated, 45.3% caveated

• HarmBench (n=150) — 98.7% base, 7.3% ablated, 48.0% caveated

• MaliciousInstruct (n=100) — 97.0% base, 0.0% ablated, 52.0% caveated

• ForbiddenQuestions (n=150) — 75.3% base, 5.3% ablated, 42.7% caveated

• SimpleSafetyTests (n=50) — 96.0% base, 18.0% ablated, 60.0% caveated — and this figure is understated. That set is mostly self-harm prompts, and the model answers them with a crisis redirect opening "I am deeply sorry to hear…", which the classifier's exact phrase list misses and scores as compliance. The real residual refusal rate on that set is higher than 18.0%. The classifier was deliberately left as-is so the numbers stay comparable with OrcaRouter's other model cards.

No reply in any set ran out of its token budget, so none of these rates is inflated by truncation. On benign prompts the same projection also removes over-refusal: XSTest-safe fell from 5.2% refusal to 0.4%, and JailbreakBench's benign subset from 25.0% to 0.0%. The published pack turns down a quarter of that benchmark's benign prompts; ablated, it turns down none.

On capability, the weights being bit-identical means there is no re-quantization to pay for, and the measurements are consistent with that:

• MMLU (n=300) — 76.7% base, 77.7% ablated, +1.0

• GSM8K (n=150) — 87.3% base, 86.0% ablated, −1.3

• CMMLU (n=500) — 76.2% base, 75.6% ablated, −0.6

Every movement is within noise at these sample sizes; a single GSM8K question is worth 0.7 points. MMLU-Pro is excluded rather than reported: its prompt asks for reasoning before the answer, and 63–64% of replies on both sides had not reached one inside the token budget, so any accuracy figure would be a floor set by the budget rather than a measurement.

The caveat that matters most

The refusal direction was estimated from the BF16 base model that the Bonsai pack was trained from. The architecture and the hidden basis are identical, so the geometry lines up. But how well that direction survives quantization-aware training has not been fully measured.

The runtime can prove, mathematically and to about 1e-6, that it removes the supplied direction from every residual write. It cannot prove from that alone that the direction still captures the same behavioural feature in the quantized model that it captured in the dense one. Those are different claims, and only the first one is settled. Anyone reading the safety table above should read it knowing that the intervention is exactly as effective as the direction-transfer assumption, and that assumption is the open question.

There is also the practical framing OrcaRouter puts on the release itself, which is worth repeating rather than paraphrasing away: removing a learned refusal direction can cause a model to respond to requests the original would have declined. This is a research and inference-control mechanism, not evidence that any resulting output is safe, correct or appropriate, and deployments using it should apply their own access controls and policy enforcement. Removing refusals is not a free improvement, and this piece is not written as though it were.

Three further practical notes for anyone reproducing it. The pack must be loaded with its own bundled runtime — an ordinary MLX loader can appear to load it successfully while silently computing the wrong thing, so if outputs look wrong before ablation is even enabled, check the loading path first. Layer-selective ablation is supported, so the intervention does not have to be all-or-nothing. And the ablation evaluation was run on the unfolded FP16 expansion of the pack rather than the pack driving its own kernels, because the packed quantized matmul has no CUDA implementation and the CPU backend needs minutes per forward pass; that expansion carries the pack's ternary values exactly and reproduces the pack's own next-token distributions to three decimals on spot checks, but it is a container change and worth knowing about. The code and the full tables are in the OrcaRouter Ternary Bonsai 2 27B Uncensored repository. A separate comparison covering the ablated MLX build against the unmodified Qwen3.8 27B MLX path goes deeper on the runtime specifics.

Where this goes, and what is still unproven

What a near-lossless 27B model in roughly six gigabytes changes for local agents is mostly about what becomes resident. A language model that fits alongside a real context window on a 16 GB laptop can stay loaded while an agent does other work — reading files, calling tools, holding a plan across turns — instead of being swapped in per request or pushed to a server. That is the difference between a local model you try and a local model you leave running, and it is the specific property that the agentic numbers, τ 2-Bench at 80.2 and BFCL v3 at 74.9, are there to support.

What is unproven is a longer list than the announcement implies.

• No independent reproduction. Every quality figure in this article — the 83.9, the 98.2%, the category averages — is Prism ML's own measurement on Prism ML's own suite. That is not a defect in the release; it is simply what one day of age looks like. It is also the first thing that will change.

• Long-horizon agentic work is the weakest part of the vendor's own table, not the strongest. Terminal-Bench 2.1 at 52.8 against 69.7 is a real gap, and the vendor says the capability is partial.

• Prompts you have not tried. The failure profile of low-bit models is selective, and IQ2_XXS's collapse on AIME26 and LiveCodeBench while holding 85.79 on MMLU-Redux is the clearest evidence available that a benchmark average does not tell you what happens on your workload. Bonsai 2 does not show that collapse on those two benchmarks, which is encouraging and not the same as a guarantee.

• The direction-transfer question in the abliterated variant, above, which is unresolved by construction.

• Whether the kernels hold up as the runtimes move. Right now this model needs a fork; stock llama.cpp rejects two of the three formats and silently garbles the third. Until those kernels land upstream, "runs anywhere llama.cpp runs" is not yet true of this model.

The release itself is not in question. A 27B-class multimodal model at 5.93 GB, at a ninth of the footprint of the thing it was compressed from, with math and coding level with the parent and instruction following slightly ahead, is a genuinely different operating point for local inference. The reasonable posture on 18 September 2026 is to take the file size as fact, take the retention figure as a careful vendor claim made one day ago on a suite the vendor chose, and reserve judgement on your own workload until you have run it on it.

The runtime-ablation code, the refusal direction and the full evaluation tables are published by OrcaRouter, alongside the routing platform the team builds.