A generated title card headed 'Karpo Decision 31B vs Nimble v3' with the subheading 'The licence answers before the leaderboard does.' Below, two cards: 'Nimble v3' listing '9B LoRA adapter', 'CC BY-NC 4.0 — no commercial use' and '51.06 Full score' beside a closed padlock; and 'Karpo Decision 31B' listing '31.3B full checkpoint', 'Apache-2.0 weights' and '73.86 Full score' beside an open padlock.
Guides & Insights

Karpo Decision 31B vs Nimble v3: The Licence Answers Before the Leaderboard Does

Author

Elias Hawthorne

Date Published

Latest models · 20View all models →
Benchmarks: Artificial Analysis · updated daily
Back to all posts

Two decision models shipped in the same autumn with almost no announcement, and they land on opposite sides of the one question that usually decides a purchase. Nimble v3 — filed as Bespoke-Nimble-9B-v3, uploaded to Hugging Face on 30 September 2026 — is a 9B LoRA adapter under CC BY-NC 4.0, which the card glosses as "it may not be used for commercial purposes." Karpo Decision 31B, created on 5 November 2026 by MachinePulse, is a 31.3B Gem​ma 4 fine-tune under Apache-2.0 with Gem​ma 4's own terms carried alongside. Everything else in this matchup — a 22-point leaderboard gap, a three-times difference in weight size, a calibration error that differs by a factor of two and a half — is downstream of that fork, and most teams discover it after they have already picked a winner on benchmarks.

So take the licence first, because it is the only part of this comparison that is both settled and decisive. Then look at what the numbers can and cannot tell you, because on that side almost nothing is settled at all.

What each repository actually contains

Nimble v3 is five files: a README, adapter_config.json, adapter_model.safetensors, a schema_config.json and .gitattributes. It is a PEFT adapter, not a standalone model, and it expects Qwen/Qwen3.5-9B as its base, which the repository does not duplicate. The card describes a context and typed questions — a choice among options, or yes or no — and returns a probability for each option. Bespoke Labs has published no context length, no option ceiling, no temperature and no change note for v3; the family's earlier checkpoints documented some of that, this one does not.

Karpo Decision 31B is a full checkpoint, 62.5 GB of weights at 31.3B parameters in bf16, plus a 255 × 5,376 fp32 readout head. It reads text, JSON and images, takes up to 255 options per question, caps a sequence at 32,768 tokens, splits longer requests by question and refuses anything that still will not fit rather than truncating it. It ships a Python package and an HTTP server in the System One shape, POST /v1/systemone, because a generic chat-completions stack cannot run it: the branch attention and the readout are both load-bearing, and loading the checkpoint as an ordinary chat model will not reproduce its answers.

The repository difference has a practical edge to it. Nimble v3's README does not tell you how to serve it correctly; the Karpo card does. One is a checkpoint with an owner's manual, the other is a checkpoint with a licence line.

The gap, and the instruments you would have to trust to believe it

On the Decision Index — the community board tracking open reproductions of TypeSafe's Je​v, now in edition 0.4.1 — the two numbers are 73.86 and 51.06 on the Full score. That is a 22.8-point gap. The Full figure is the one worth trusting most, because the maintainers ran it themselves: roughly 60% of it comes from tests over the same skills as the public suite and 40% from domains nobody published for, and neither vendor saw either half beforehand. Karpo's split is 70.94 same-skills against 61.5 new-domains. Nimble v3's is 52.34 against 38.2.

The public figures on each card are the vendors' own runs of the official kit and should be read separately: 75.05 for Karpo across 140,178 scored requests, and 56.88 for Nimble v3 — the figure its card prints on the older 0.2.1 suite, which the board's rerun puts at 57.19 on the current scale. Both were adopted by the board, which is a real review step and not the same thing as an independent lab putting the model on its own hardware. Neither model appears on Artificial Analysis. Nobody outside these two repositories has scored either one.

What that means in practice is that the 22.8 points are more trustworthy than most vendor comparisons and less trustworthy than a single instrument reading both models. They were produced by one organisation on one suite with one formula, which is genuinely the best case available for open decision models — and the suite is unofficial, the tie band is 0.9 points, and the private half cannot be inspected by you. Treat the direction as real and the magnitude as approximate.

Where the 9B model is closer than the size gap suggests

A generated dual-bar chart titled 'Five category scores, one board' with rows reading 'Tools and automation — 84.24 / 88.39', 'Language understanding — 60.68 / 76.78', 'Retrieval — 55.48 / 76.42', 'Knowledge and reasoning — 40.96 / 65.46' and 'Arts and open-ended — 44.05 / 68.21', each pairing a shorter blue Nimble v3 bar with a longer cyan Karpo Decision 31B bar. The top row carries a warm outline and a tag reading 'closest'. The footer reads 'Decision Index 0.4.1 category scores; both models scored by the same maintainers on the same suite.'

Split the board's five category scores and the story stops being a walkover. On tools and automation, Nimble v3 scores 84.24 against Karpo's 88.39 — four points, from a model a third the size, on the category that covers tool selection and agent gating. That is the use case Bespoke Labs built the family for, and it survives the size difference more or less intact. If your decision layer exists to answer "should this agent call the tool, and which one," the 9B adapter is in the same conversation as the 31B checkpoint.

Everywhere else the gap is the one you would predict. Language understanding runs 60.68 against 76.78, retrieval 55.48 against 76.42, knowledge 40.96 against 65.46, arts and open-ended judgement 44.05 against 68.21. Karpo's advantage is concentrated in exactly the places a 31B model's extra capacity should pay: longer context, more world knowledge, better reading of a dense state.

Latency does not follow the size gap either. On the tracker's reference hardware — one GPU, one request at a time — Nimble v3 records a 73.9 ms median against Karpo's 94.3 ms. Twenty milliseconds is not a reason to choose a model, but it is a reminder that a merged 9B adapter is a different object to operate: it fits on a laptop in Q4, it loads in seconds, and it can sit inside a per-step agent loop without a GPU budget conversation.

Calibration, where the smaller model is not merely weaker but less honest

Accuracy tells you how often the top answer is right. Calibration tells you whether the probability attached to it can carry a threshold, and for a decision model that is the only number that matters operationally.

Karpo's card reports a 4.8-point expected calibration error across 34,411 responses: mean confidence 87.96% against an actual accuracy of 83.19%. In the top bucket, which holds 68.5% of all responses, it is right 95.7% of the time while claiming 98.7%. Overconfident at the top end, by about three points, with the curve otherwise tracking.

Nimble v3's is 12.29 points across 34,833 responses: confidence 87.26% against accuracy 74.97%. The top bucket holds 67% of responses, claims 98.6% and is right 89.1% — a 9.5-point overstatement in the band where a threshold actually lives. Below that, the mid-range buckets are worse still: at a claimed 0.85 the model is right about 59% of the time. That is not a model with a small calibration drift; it is a model whose confidence cannot be used as a gate without rescaling, and the family's own documentation never promised otherwise.

Read those two paragraphs together and the honest conclusion is narrower than the leaderboard suggests. Karpo is better at decisions and better at knowing how good its decisions are. Nimble v3 is cheaper to run, and on the one category that matters most to agent work it is close — but if you plan to auto-accept above 0.9 and escalate below, you will spend your first week fitting a correction to Nimble that Karpo gives you in its card.

The licence fork, stated without hedging

Nimble v3 is CC BY-NC 4.0. That is not ambiguous: the card says it may not be used for commercial purposes. Every earlier Nimble checkpoint was Apache-2.0, and the base model is Apache-2.0, which changes nothing — a permissive base does not loosen a non-commercial adapter, and the adapter is where the behaviour lives. The GitHub repository that holds the project's training and curation code reports no licence at all through GitHub's API, which is a separate and arguably worse problem for anyone planning to build on it.

Karpo Decision 31B is Apache-2.0 for weights and code, with Gem​ma 4's terms carried alongside in the repository. That is the permissive end of open, with the usual Gemma caveat that downstream redistribution has to carry those terms too.

For a research notebook, an internal evaluation, or a paper, the distinction is invisible and Nimble v3 is the cheaper entry. For anything that touches revenue — a routing gate inside a commercial product, a triage model in a paid pipeline, even an internal tool at a for-profit company — the licence settles the matchup before the 22.8 points get a hearing. Bespoke Labs has a hosted API path and a placeholder for a 27B variant that reports 65.33 on the older public suite, and neither changes the terms on the 9B adapter.

Serving either one is a routing problem before it is a modelling problem

A generated architecture diagram titled 'The decision head and the generative half'. A left card labelled 'Decision model, self-hosted' lists 'Karpo Decision 31B — Apache-2.0', 'Nimble v3 — CC BY-NC 4.0' and 'answers only the options you supply', and an arrow runs right into a centre card labelled 'One OpenAI-compatible key' listing '200+ models', 'provider list price, no markup' and 'automatic failover'. Two arrows fan out from the centre to cards reading 'TypeSafe Jev 1.14' and 'Gemma 4 31B'. The footer reads 'Neither decision model is hosted here; they stay where their licence permits.'

Decision models are narrow by construction: they answer only the questions you hand them, they cannot write, explain or return an option you did not supply. That makes them the small half of a two-model pattern, and the half that is easiest to keep in-house. The generative half is where a single endpoint earns its keep.

OrcaRouter runs neither of these checkpoints, and neither would make sense behind a chat-completions route. The two models it does serve sit one layer down: TypeSafe's Je​v 1.14, the target the whole board exists to reproduce, and the Gem​ma 4 31B checkpoint that Karpo was fine-tuned from. One key covers both, pricing follows the provider's list with nothing added on top, so a vendor change reaches you the same day. What suits a decision head is the routing layer in front of it — a routing DSL that sends each request class to the model that should answer it, with failover behind that — so a nightly rescoring job and a live guardrail share one contract while the decision model itself stays wherever its licence allows.

Three things that would change this reading

A generated checklist card titled 'Three things that would change this reading', with three numbered rows: 'A commercially usable Nimble adapter' — 'Apache-2.0 on a Qwen3.5-9B base, with a context contract'; 'A published context and prompt spec' — 'Nimble v3 documents neither, so its inputs are reconstructed'; and 'One instrument scoring both' — 'A single board reading both models on the same suite'. The footer reads 'Until then: Apache-2.0 and better-calibrated at 31B, or non-commercial and cheap at 9B.'

The first is a licence change. A commercially usable Nimble adapter on a Qwen3.5-9B base, with a context contract and a stated temperature, would make the 9B option genuinely competitive for agent gating — the 84.24 tools score says the capability is already there. Nothing in Bespoke Labs' public material suggests that is coming; the move from Apache-2.0 to CC BY-NC went the other way.

The second is a context and prompt specification. Nimble v3's card documents neither, which means anyone running it is guessing at a prompt format that the earlier releases used to ship. A model whose inputs are reconstructed by the user is a model whose numbers are hard to reproduce, and the 12.3-point calibration error is the kind of figure that gets better or worse depending on how you frame the question.

The third is any independent instrument reading both. The Decision Index is careful, transparent about its method and honest about its tie bands, and it is still one community board. The day either model appears on Artificial Analysis next to a hyperscaler scorer, the comparison stops being arithmetic across two vendor cards and starts being a measurement. That has not happened for either, and until it does the defensible position is the boring one: Apache-2.0 and better-calibrated at 31B if you can run it, non-commercial and cheap at 9B if you cannot, and no reason to pick on the scoreline alone.