A generated title card headed 'Karpo Decision 31B vs Kev 27B' with the subheading 'Same board, fourteen points apart — and the loser has the better calibration', four chips reading 'Karpo 73.9 Full', 'Kev 59.6 Full', 'calibration error 4.8 vs 2.18' and 'identical private tests', and two outlined cards labelled 'Kev 27B' and 'Karpo Decision 31B' joined by a line reading 'both graded by the Decision Index'.
Guides & Insights

Karpo Decision 31B vs Kev 27B: The Better-Documented Model Loses by 14 Points

Author

Elias Hawthorne

Date Published

Latest models · 20View all models →
Benchmarks: Artificial Analysis · updated daily
Back to all posts

This is the rare matchup in which both models were graded on the same instrument by the same people. Karpo Decision 31B and Kev 27B both sit on the Decision Index, the community board that scores open reproductions of TypeSafe's Jev, and the gap between them is not small: 73.9 Full for the MachinePulse model against 59.6 for Jared Palmer's Kev 27B, a spread of fourteen points on identical private tests. Karpo leads the board. Kev is 74th of 307.

That result is awkward, because Kev 27B is the more rigorously documented model by a wide margin, and on the one property a decision model is actually hired for — the honesty of its probabilities — it is measurably better than the model that beats it. Both of those claims come from the same two sources, and understanding why they do not contradict each other is the most useful thing in this comparison.

What each model is, before the scores

Kev 27B is a full-weight fine-tune of Qwen3.8-27B published by Jared Palmer under Apache-2.0. Its second version, released on 30 September 2026, is a weighted average: every backbone tensor is 0.85 × the fine-tuned weight plus 0.15 × the previous version's, with the pointer head kept from the new run. The served checkpoint is 25.6B in the backbone plus a 2.6M-parameter head, 51.3 GB in bf16, with a single calibration temperature of 1.32 baked into the head file. States run to 65,536 tokens with at least 8,192 per question, and that figure is validated rather than nominal — the card publishes the paired accuracy difference at every bucket from 8k to 64k against the 8k baseline, with confidence intervals.

Karpo Decision 31B is a fine-tune of Gemma 4 31B instruction-tuned, published by MachinePulse on 5 November 2026, also Apache-2.0. It keeps 31.3B parameters in bf16 plus a 255 × 5,376 fp32 readout — 62.5 GB on disk — and answers through a linear head over option codes rather than a pointer head over tokens. Context is 32,768 tokens per sequence, split by question when a request overflows, with an outright refusal rather than truncation when a single question will not fit.

Same job, different engineering. Kev reads a long document and produces a distribution. Karpo reads a somewhat shorter one and produces a distribution, faster.

The 14 points, and what produced them

A generated two-column scoreboard titled 'Karpo Decision 31B vs Kev 27B — the scoreboard'. The Karpo column reads 'Full score: 73.9', 'Rank: first of 307, tied', 'Calibration error: 4.77 points', 'Context: 32,768 tokens', 'Latency: 94 ms median' and 'Licence: Apache-2.0'; the Kev column reads 'Full score: 59.6', 'Rank: 74th of 307', 'Calibration error: 2.18 points', 'Context: 65,536 tokens', 'Latency: 135 ms median' and 'Licence: Apache-2.0'. The footer reads 'Scores and calibration per the Decision Index; latency per each model card, hardware not identical.'

Both rows come from the same board, so this section is one of the few places where a direct subtraction is legitimate.

• Full score — Karpo 73.9, Kev 59.6.

• Public index — Karpo 75.05, Kev 56.69.

• Rank by Full score — Karpo 1st (tied with GLiDE 27B at 73.4), Kev 74th of 307.

• Knowledge and reasoning — Karpo 65.5 chance-corrected skill, Kev 45.7.

• Tools and automation — Karpo 88.4, Kev 72.7.

• Arts and human taste — Karpo 68.2, Kev 42.7.

• Median latency, one GPU, one request at a time — Karpo 94 ms, Kev 135 ms.

The shape of the loss is consistent across every category, and the largest absolute gap is in knowledge, at nearly twenty points. That is the one that should worry anybody planning to use Kev for document decisions: the model was trained heavily on document and developer-tooling work — 24,000 of its 145,840 training records come from licensed task families, another 7,617 are assembled long documents — and it still lands two-thirds of Karpo's knowledge skill.

Kev's own card is candid about the direction of travel. Measured on its own breadth suite against TypeSafe's Jev, it reports an index of 52.3 with a confidence interval of [49.2, 55.4] against Jev's 54.0 [51.2, 57.0] — a loss it states plainly and does not dress up. A team that publishes its own losing interval is a team whose other numbers deserve the benefit of the doubt. That is precisely what makes the 59.6 credible rather than embarrassing.

The number that runs the other way

Here is the twist. Calibration error — how far the stated probability drifts from the observed hit rate — favours Kev, and not narrowly.

The tracker records an expected calibration error of 2.18 points for Kev 27B against 4.77 for Karpo Decision 31B. In plain terms: when Kev says 0.8, it is right about 80% of the time, with a two-point average miss. When Karpo says 0.8, the average miss is closer to five points. Kev's card corroborates this from its own side, publishing per-panel ECE figures between 0.014 and 0.053 and a five-fold cross-validation that lowers fitting error from 4.8% to 3.8%.

So the model that is worse at answering is better at knowing how good its answers are. That is not a paradox, it is the distinction between accuracy and calibration, and it decides which model belongs where:

• If the decision is automated at a threshold, calibration is the governing property. A gate set at 0.9 that fires on Karpo's 0.9 answers will be wrong about twice as often as the same gate on Kev's. Kev's lower accuracy costs you throughput — more cases fall below the threshold and go to review — and its better calibration means the ones that clear the bar are safer.

• If the decision is reviewed or scored, accuracy dominates. A human or a downstream process sees the top answer and the probability, and fourteen points of accuracy is fourteen points of work done correctly. Kev's calibration advantage buys nothing when nobody is thresholding.

A capture of the community Decision Index leaderboard, headed with the multimodalart jev-decision-index attribution, listing Karpo Decision 31B at 73.86 in first place ahead of GLiDE 27B at 73.37, REAX 1.3 at 72.64, Logits Sigma-1 Duo at 72.28, Copernicus-31B at 72.11, KnowLine 31B Gen1.5 at 71.91, Copernicus-57B at 71.8, FlexDM-LoRA 2.1 at 71.66, Sieve 31B v3 at 71.10, OneJev1.6g-31B at 70.86 and Turion Jev 31B at 70.47.

Both models deliver the same three question types — choice up to 255 options, noul for true/false, score for an ordered rubric — and both expose a System One-shaped HTTP endpoint, so the choice is reversible in code. That is the good news: you can measure this on your own data in an afternoon rather than argue about it.

Document length is where the specs actually diverge

Karpo's 32,768-token ceiling is per sequence, and the split rule is "by question, each part carrying the full state". For a hundred questions over a 20k-token contract, that is one pass and no drama. For one question over a 40k-token contract, the request is refused — nine of the 140,178 requests in MachinePulse's own run failed exactly this way.

Kev serves 65,536-token states, validated in buckets, and its long-document panel is the most useful thing on the card for anyone with real contracts. On CUAD commercial agreements it reports accuracy of 0.874 across 2,194 questions, with the per-bucket figures holding between 0.867 and 0.880 from under 8k all the way to 32k–64k tokens. The accuracy does not collapse with length. What does shift is calibration: error on that long-document panel is 0.053, roughly three times its short-document figure. Long states are where Kev's confidence gets loose.

So the honest trade on length: Kev will take a document Karpo refuses, and it will answer it slightly less reliably than it answers a short one. Neither model will silently truncate, which matters more than either number — a decision model that quietly drops the clause you asked about is worse than one that says no.

Deployment, weights and the licence both share

Both models are Apache-2.0 for weights and head, both require their own loader, and both fail badly if you treat them as chat models. Kev needs an 80 GB-class GPU — a 64k-token state peaked at 87.1 GB on an H200 — or a Mac with roughly 96 GB. Karpo needs a GPU that holds 62.5 GB of bf16 weights. Neither has a hosted endpoint from its publisher; Kev is served by its own repository's server, Karpo by its own package. Both have attracted community quantisations, which is the usual signal that people are actually running them rather than reading about them.

The 135 ms versus 94 ms difference is per request on hardware that is not identical, so treat it as directional. What is not directional is the operational shape: Kev's non-causal attention over the full sequence costs more per token as states get longer, while Karpo computes one shared state per request and branches questions off it. If your workload is many questions over a short state, Karpo is built for exactly that. If it is one question over a long document, Kev is.

Which one to run

Pick Kev 27B if you need to hand a model a 64k-token agreement, if you want to threshold on probability and care that the threshold means what it says, and if you value a vendor who publishes confidence intervals on his own losses. You are accepting roughly fourteen points of accuracy on a board that both models competed on, and you are getting the field's best evidence for what that accuracy does and does not cover.

Pick Karpo Decision 31B if the state is a screenshot, a ticket or a page — short, many questions, latency-sensitive — and if the board's Full score is the evidence you weight. You are accepting a five-point calibration error and a 32k ceiling, and you are getting the current leader at 94 ms.

What neither gives you is a production endpoint. If the decision layer has to sit behind a gate that fires before every tool call, you will be running one of these yourself, or you will be calling something hosted. TypeSafe's Jev 1.14 is on our catalogue for exactly that role, and it is the model both checkpoints were built to chase — useful as the control in a shadow test, since an open reproduction that beats its target on a third-party board should also beat it on your traffic. The catalogue also carries both base models, Gemma 4 31B and Qwen3.8-27B, through one API for 200+ models at provider list price with no markup, which is the cheapest way to establish whether a decision layer is worth owning at all before you buy the GPU.

The open question this matchup leaves is the one the board cannot answer: whether Karpo's fourteen-point lead survives contact with a real workload, or whether it is the kind of lead that comes from being tuned on a distribution closer to the private test set than its competitor's. Kev's card documents its training mix down to the record count; Karpo's does not. That asymmetry is not evidence of anything, and it is the reason to measure rather than to trust either number.

A capture of the Hugging Face model card for jaredpalmer/kev-27b showing the Text Classification tag, the qwen3.8 and typesafe tags, the apache-2.0 licence, the summary describing a decision model that returns a calibrated probability distribution in a single forward pass over documents of up to 64k tokens, 'a full-weight fine-tune of Qwen3.8-27B averaged with version 1', the model-details block listing a Qwen3.8-27B backbone of 64 layers with 48 Gated DeltaNet and 16 full-attention layers, 25.6B backbone parameters plus 2.6M in the head, bf16 precision in a 51.3 GB checkpoint, and states of up to 65,536 tokens.