Nemotron 3.5 Bolt 24GB vs Nemotron 3.5 Bolt 74B: One Model, Two Artifacts

This is a comparison between a model and itself. N​VIDIA N​emotron 3.5 Bolt 74B-A9B 24GB and N​VIDIA N​emotron 3.5 Bolt 74B-A9B BF16 are the same architecture at the same parameter count, produced in the same week, from the same parent — and N​VIDIA's own documentation says plainly that one is for running and the other is for changing. Treating them as rivals is a category error; treating them as interchangeable is an expensive one. Here is the difference, measured in bytes and in benchmark points.

What the two artifacts actually are

The names describe the pipeline, not two designs. N​VIDIA took N​emotron 3.5 Super VL 120B-A12B, ran a router-guided architecture search called Star Elastic against a memory budget, and produced a 74-billion-parameter model with 9 billion active parameters. That result exists in two published forms: the BF16 weights as they came out of post-training, and the same weights with the expert layers quantised to 2.0625 bits per element and repaired with quantisation-aware distillation.

Both are dated 15 October 2026 on their model cards. Both carry the same OpenMDW-1.1 licence, the same 256,000-token context window, the same Mamba-2 hybrid LatentMoE backbone and the same C-RADIO v4-H vision encoder. The repository metadata makes the relationship explicit in a single field: the 24GB checkpoint's card lists base_model_relation: quantized against the BF16 release.

What differs is what each one is for. The 24GB card names its use case as local agent harnesses — tool calling, browsing, terminal work — and its deployment as vLLM on Ampere, Hopper or Blackwell. The BF16 card names its use case as post-training with NeMo RL and NeMo Gym, domain adaptation, distillation, and producing your own quantised variants, and warns in the same paragraph that for latency- and throughput-optimised inference you should use the 24GB version instead. A vendor telling you not to run a checkpoint you just downloaded is worth reading literally.

The size difference, read from the file listings

• BF16 — 149.8GB total across 63 safetensors shards, largest shard 3.71GB, plus a 228MB vision tower.

• 24GB safetensors — 24.12GB across 90 shards in the vLLM-ready build, which is marginally above the number in the model's name.

• 24GB GGUF — 22.02GB in a single file, plus a separate 1.73GB vision projector, for 23.75GB in total.

• Ratio — roughly six to one in bytes, from 16-bit experts to 2-bit experts plus mixed-precision treatment of everything else.

Sixty-three shards against ninety is also a shape difference, not just a size one. The BF16 release is built to be loaded by a training framework across whatever parallel layout you have; the 24GB release is built to be loaded by one GPU, or by llama.cpp on one machine, and the shard count reflects the different job.

Three gigabytes of the 24GB spread is worth understanding before you plan storage. The vision projector is a separate file in the GGUF build because that is how llama.cpp handles multimodal input, and anyone who downloads only the weights and then wonders why images do not work has skipped it. In the safetensors build the vision tower is inside the shards and the total crosses the 24GB line.

What the quantisation cost, on N​VIDIA's own numbers

The BF16 card publishes both columns side by side, which is unusually forthcoming: the same benchmark table carries the IQ2 24GB figures and the BF16 figures for the same thirteen evaluations. Vendor-reported, unreproduced, and still the most direct evidence available about what two bits per expert costs.

• Terminal-Bench 2.1 — 50.98 at IQ2 against 53.93 at BF16, a loss of 2.95 points.

• BrowseComp — 66.75 against 70.00, a loss of 3.25.

• GPQA Diamond — 86.43 against 88.83, a loss of 2.40.

• AA-LCR long-context reasoning — 64.06 against 67.38, a loss of 3.32.

• HLE without tools — 25.30 against 28.73, a loss of 3.43.

• AA-Omniscience accuracy — 16.90 against 21.80, a loss of 4.90, the largest in the table.

• PinchBench — 87.82 against 90.35, a loss of 2.53.

• GDPVal-AA-V2 — 28.47 against 31.13, a loss of 2.66.

Two rows break the pattern, and they are the interesting ones. On AA-Omniscience non-hallucination the quantised model scores higher, 61.27 against 56.14 — compressing the experts made the model less likely to answer confidently when it does not know, which is the kind of finding that makes a table worth publishing in full. On WebArena vision the two are level, 50.86 against 50.25, suggesting the vision path survived the compression more or less intact, and the vision adapter retraining described in the model card is presumably why.

Read together: a two-to-five-point tax on most capability rows, near-zero on visual grounding, and an improvement on one calibration metric. That is a favourable outcome for a six-to-one compression, and it is still a tax you should count rather than wave away.

Which one you can actually serve

The 24GB artifact has a runtime story and the BF16 one has a training story, and they are not the same story with a different flag.

The 24GB GGUF is listed for llama.cpp, and N​VIDIA points the same model at Ollama, LM Studio and Unsloth. The vLLM path is stricter than it sounds: the reference recipe pins a nightly container by commit hash, requires a separate kernel package fetched at runtime, sets the v2 model runner explicitly, names b12x for the linear and MoE backends with FlashInfer for attention and Mamba, pins the KV cache to fp8, and sets max-num-seqs 1. That last flag is the one to notice — the published serving configuration is a single concurrent sequence, which is a laptop-shaped configuration, not a serving-shaped one.

The drafter is a third download. LiLiCorr is a 1.13B companion checkpoint whose card states in as many words that it is not a standalone language model, that it proposes draft tokens and the 24GB target verifies them, and that the target produces the visible text. N​VIDIA's reported 83.99 tokens per second on AIPerf SPEED-Bench-Coding assumes it is attached, with seven drafted tokens per step and mean acceptance lengths of 4.68 under greedy decoding and 4.25 with sampling at concurrency 64.

The BF16 checkpoint's serving story is your problem to solve. It needs roughly six times the memory of the quantised build before any KV cache, which puts a single 80GB accelerator out of reach and makes multi-GPU or multi-node the default assumption. That is the right trade for post-training and the wrong trade for inference, which is exactly what the card says.

The decision, stated plainly

If you want to run an agent today on hardware you own, download the 24GB build, accept the two-to-five-point tax, and budget for the drafter and the projector as well as the weights.

If you want to change the model — fine-tune on your own data, distil it into something smaller, or compress it differently for hardware N​VIDIA did not optimise for — download the BF16 build and treat the 24GB release as documentation of one possible outcome.

The interesting middle case is the one the BF16 card invites explicitly: producing your own quantised variant. Someone with a different accelerator, or an appetite for four-bit weights, or a workload that needs more concurrency than the reference recipe allows, can take the full-precision weights and build the artifact N​VIDIA did not. The quantisation tax above is then the number to beat, and it is now a published target rather than a mystery.

Where a gateway fits into a local stack

Neither of these checkpoints is hosted anywhere, including by us — the OrcaRouter catalogue carries no N​emotron model, so there is no endpoint and no price to quote for either. Local inference is also only ever half of an agent architecture: the machine that runs Bolt still needs to reach models it cannot download, whether that is a frontier model for the hard cases, a vision model for something Bolt handles badly, or a second opinion on a decision that matters.

That is the layer we operate. One key across 200-plus hosted models at the provider's list price with no markup added; automatic failover when a provider degrades mid-request; a routing DSL that turns "try Bolt locally, escalate to a hosted model when the task exceeds it" into configuration rather than code. The 24GB half of the stack is decided by your hardware and your tolerance for a nightly container. The half on the other side of the escalation is decided by whoever holds the key.

What would change this picture

Two things, and both are predictable.

The first is a four-bit or FP4 build of Bolt from N​VIDIA or from the community, given that the parent family already ships NVFP4 checkpoints and the BF16 release exists precisely so that such variants can be made. A four-bit Bolt that recovered half the gap above while still fitting in a 24GB envelope would make the quantisation tax argument considerably less interesting.

The second is any independent evaluation of either artifact. The Artificial Analysis panel has measured the 24GB model at 21 on its Intelligence Index with no speed and no cost published, because there is nothing to measure a price against. Nobody has published a reproducible run of the BF16 checkpoint at all, which is unsurprising for a 150GB download intended for training. Until someone does, the gap between the two columns above is a vendor's account of its own compression, and the honest way to read it is as a claim that happens to be unusually well supported by the shape of the evidence.

A result like this is also a reminder that model mistakes are already documented: the AI incidents and agent failures behind models in this class are collected in the OrcaRouter AI Incident Archive, where each entry carries its own sources, severity, confidence grade and disputed flag.