Gemini Decision 1 vs Kev: A Thousand Questions in One Call, or a Recipe You Can Retrain

Jared Palmer's Kev family cost about $95 in H100 time on Modal to train, plus roughly three cents of API calls for evaluation data, and the whole recipe is in the repository. Gemini Decision 1 cost whatever a frontier lab spends on a model it will not describe, and what you get for the money is a request schema: one endpoint, up to 1,000 typed questions against a single 262,144-token input, calibrated probabilities back, zero output tokens, $0.10 per million input tokens. Put Kev next to it and the comparison is not really between two models. It is between a batch-scoring appliance and a small piece of machinery you can take apart.

Neither project has published a number the other one could be scored against, which is the first honest thing to say. Google shipped no benchmark table with Gemini Decision 1; every performance figure in this piece comes from Artificial Analysis, which has the model at 66 on Decision Index v2.1. Kev does not appear on that board at all. What Kev publishes instead is a harness, a locked test set and its own error rates, and those cannot be subtracted from anything.

The thing Gemini Decision 1 does that Kev cannot

Start with the capability gap, because it is the one that decides architectures rather than scores. Gemini Decision 1 evaluates up to 1,000 independent questions against one shared input in a single forward pass. The questions cannot see each other, adding or removing one does not change the others, and the shared input is billed once.

That last clause is the product. A support ticket checked against forty policy rules is one request, one read of the ticket, forty calibrated answers. On a per-question model — which is what Kev is, in the sense that the head scores each option against the question in one row — the same forty checks are forty passes over the state, whether you batch them or not. Kev's own serving figures make the cost concrete: 18.1 milliseconds for six questions over a short state on an H100, 12.9 milliseconds cached, 14.3 GB resident. Six questions, not a thousand.

The input surface differs the same way:

• Modality — Gemini Decision 1 reads text, image, video, audio and PDF; Kev reads text. A call recording or a scanned form is a preprocessing problem for Kev and a native input for Gemini Decision 1.

• Context — 262,144 tokens for Gemini Decision 1 across input and questions; Kev serves up to 65,536 with at least 8,192 per question but validates only up to 8,192, and its card says so.

• Abstention — Gemini Decision 1 returns probabilities bounded to 0.01–0.99 with a confidence field, so a low-confidence answer is visible without a separate option. Kev returns a softmax over the options you supply, which means abstention is whatever option you name plus a threshold you set.

• Weights — none for Gemini Decision 1; a hosted endpoint on the Gemini Developer API at v1beta, with a free tier of 200 requests per minute and 10,000 per day. Apache-2.0 for Kev, with a rank-16 LoRA adapter on a frozen Qwen3.5 base and a pointer head, in 0.8B, 4B and 9B sizes.

The thing Kev does that Gemini Decision 1 cannot

Kev lets you change the model. Not prompt it differently — retrain the decision behaviour on your own labels, for an amount of money you can put on an expense report.

The construction is deliberately modest: a frozen Qwen3.5 base, a rank-16 adapter of about 33.8 million trainable parameters, and a small pointer head that scores each option's closing token against the question's final token. Because the base stays frozen, the marginal cost of a new decision domain is adapter-sized, not foundation-model-sized. You can hold one base in memory and swap adapters per domain, which is a deployment shape no hosted endpoint expresses.

What Kev publishes about itself is specific enough to argue with, which is rarer than it should be:

• Transfer-v4, locked out-of-domain — accuracy 0.838, Brier 0.224, ECE 0.017.

• Breadth-v1, 14 held-out datasets, 3,089 questions — accuracy 0.690, ECE 0.029.

• Documents-v1 — 0.903. Hard-v1 — 0.803. MMLU-Pro with ten options — 0.565.

• Date arithmetic — 0.65 at day precision, named by the author as the weakest area, with a preprocessor flag offered as a partial fix and no re-measurement claimed.

Those are the author's own measurements on the author's own harness, and the card says the calibration is a single in-distribution temperature, so coverage degrades out of domain and option order can flip an answer. That is a credible self-assessment, not an audited result. It is still a scale you can try to reproduce tonight, which is more than you can say for most vendor numbers in this category.

Why the two numbers on the table never meet

The temptation is to put Kev's 0.838 transfer accuracy beside Gemini Decision 1's 66 and announce a winner. Resist it, for three reasons.

The instruments measure different things. The Decision Index scores accuracy minus the Brier penalty for the confidence attached to the answer, aggregated across six evaluations, 45% of which is built on private data. Kev's transfer-v4 number is plain accuracy on a locked set the author assembled, with Brier and ECE reported beside it rather than folded into the headline.

The task mixes are not the same. The index's largest component, AA-Paperclip at 30%, is fifteen all-or-nothing answers per document — a shape Kev's documents-v1 test approximates but does not replicate.

And the denominators differ. An index score is a weighted composite where abstention is rewarded; an accuracy figure is the share of questions where the top-probability option was correct. A model can beat another on one and lose on the other without either being wrong.

So the defensible statement is narrow: Gemini Decision 1 is mid-table among hosted decision models at 66, ninth of eighteen, with a first-place finish on the knowledge components and a 33% all-or-nothing document pass rate; Kev is a 4B-class adapter whose author reports strong in-domain calibration and weak date arithmetic. Anyone who tells you one is better is extrapolating.

What each one costs to actually run

Gemini Decision 1's economics are easy to compute and hard to like at volume. At $0.10 per million input tokens, Artificial Analysis measures $345 per million tasks — the most expensive model in the top half of the board, with no context caching, no Batch API and no Flex or Priority inference. Repeated shared input counts in full on every request. A nightly sweep of a million tickets re-reads the policy document a million times.

Kev's economics are the opposite shape: nearly free per call once the GPU is paid for, with the real cost sitting in the hours you spend building labels, training adapters and validating them. Palmer's ~$95 figure covers the training run, not the labelling. If your routing rules differ from a generic judge's — and they do — that labelling cost is buying you the thing no hosted decision model sells: a scorer that encodes your policy rather than approximating it.

There is a third cost that decides more projects than either price: iteration speed. Gemini Decision 1's limits are documented and fixed, so a behaviour change is a prompt rewrite and a re-run of your evaluation set. Kev's is an adapter retrain, which is cheap in compute and not cheap in process. For a team with one domain and high volume, the retrain pays off. For a team with twelve domains and no ML engineer, the hosted endpoint wins on engineering hours even at $345 per million tasks.

If the pipeline around the decision is where you actually spend

OrcaRouter carries neither model. Kev is Apache-2.0 weights you run yourself, and Gemini Decision 1 is a probabilities-only endpoint on Google's own API — neither is a chat-completions target, and we do not pretend otherwise. The decision model in our catalogue is TypeSafe's typesafe/jev-1.14, at 66.23 on the same board, for teams that want a scoring call on the same key as everything else.

The half of the pipeline we are built for is the one upstream of the decision: generating the candidate answer, writing the rubric, summarising the ticket before it is scored. That call is where you change models most often and where a second vendor contract costs the most, and it is the one that runs across a single endpoint with more than 200 models on it, provider list price passed through at 0% markup, and automatic failover when a provider degrades. If you are weighing Kev against a hosted scorer, run the drafting half through us and keep the scoring decision reversible — that way the choice you make today about a $95 adapter is not a migration later.

The decision between the two decisions

Choose Gemini Decision 1 when the decision is one input and many rules, when the input is audio or video or a PDF, when a thousand checks in a single billed pass changes your cost model, and when you are content to trust a calibration you did not fit. Accept that you are buying a black box at frontier-lab input pricing with no batch discount.

Choose Kev when the decision is one input and a few typed questions, when the text is short enough to validate inside 8,192 tokens, when your policy is specific enough that a generic judge approximates it badly, and when owning the artefact matters more than owning the endpoint. Accept that you are now responsible for the calibration, the date arithmetic and the option-order sensitivity the card warns you about.

The one thing this comparison should not do is produce a league table. There is no common instrument, no shared task set and no reproduction of either number by a third party. What you can do this week is put 200 of your own labelled items through both and compare pass rates and confidence curves — a morning's work that will tell you more than the next leaderboard revision will.