Solar Open 2 250B: Frontier Agent Scores With Nowhere to Run Them

The Hugging Face card for Solar Open 2 250B 250B hands you a serving command, and the command does not work. It asks vLLM for a reasoning parser named solar_open2, a tool-call parser named solar_open2, and a logits processor called SolarOpen2TemplateLogitsProcessor — three things that exist in no released version of vLLM. Support for the architecture showed up in mainline as an unmerged pull request on August 5, two weeks after Upstage published the weights on July 22. That gap is still the most interesting fact about Solar Open 2 right now, and it is the part every launch write-up skipped in favour of the benchmark table. So here is the other version: what Upstage actually shipped, what its own numbers do and do not establish, what the one external score now on the record changes, and what it takes to get a single token out of the thing — measured against DeepSeek V4 Flash, the model Upstage picked as its main yardstick.

What Upstage actually shipped

Everything below comes from the repository's own config.json and the Solar Open 2 technical report (arXiv 2607.20062) — primary sources, but the vendor's primary sources. No independent lab has published its own measurements of this model. One external figure has now surfaced, a press-reported Artificial Analysis Intelligence Index score of 37 from the government's second-round evaluation, and it gets its own paragraph in the benchmark section below because it is reported, not reproduced.

• Size — 250,287,794,944 parameters total, roughly 15B active per token

• Sparsity — 320 routed experts plus 1 shared expert, top-8 routed per token, and first_k_dense_replace set to 0, meaning the mixture starts at layer zero with none of the dense warm-up layers that comparable MoE designs usually keep

• Shape — 48 layers, hidden size 4096, head dim 128, 64 query heads against 8 key/value heads

• Attention — 12 softmax (GQA) layers, at indices 0, 4, 8 and so on to 44; the other 36 layers are linear attention. Three quarters of the stack has no softmax in it

• Positions — use_rope is false. There is no positional encoding at all

• Context — max_position_embeddings of 1,048,576

• Vocabulary — 196,608 tokens, across English, Korean and Japanese

• Weights — bfloat16, which works out to 500.6 GB. That is just 250.29 billion parameters times two bytes, and it matches the footprint Nota AI reports for the unquantized model

• Training — about 11.9T tokens per the report: 10T general pre-training, 1T on a curated high-quality subset, 0.9T for the length extension, at a stated 4:6 real-to-synthetic ratio with English above 80% and math and code each at least 15%

• License — the Upstage Solar License, which has strings attached (below)

The predecessor, Solar Open 1, was a 100B model released in January. Solar Open 2 is not a refresh of it so much as a different animal.

The architecture is the actual story

The layer pattern is one softmax layer followed by three linear-attention layers, repeated twelve times. That much has been reported. What has not is which linear attention, and why it matters.

The linear layers are KDA — Kimi Delta Attention, the mechanism Moonshot AI introduced for Kimi Linear. Upstage did not invent a new linear attention; it adopted someone else's and changed one thing. The change is visible in the config as kda_allow_neg_eigval set to true, and the report explains what it buys: standard KDA constrains the state-transition eigenvalues to the range [0,1], so each step can only add to the recurrent state or decay it. Setting that flag widens the write strength to beta = 2·sigmoid(·), a range of (0,2), which pushes the eigenvalues out to [-1,1]. A step can now flip the sign of what is already stored. In plain terms: the state can correct itself rather than only accumulate, which is Upstage's stated mechanism for holding a long agent trajectory together across many tool calls.

The dozen softmax layers are GQA with an elementwise sigmoid output gate — use_gqa_gate is true in the config — which the report credits with adding non-linearity and suppressing attention sinks.

And the reason there is no RoPE follows from the linear layers. Because a recurrent state encodes token order intrinsically, the report argues you can delete positional encoding entirely and stop being bound to the length distribution you trained on. RoPE, in this framing, is the thing that caps extrapolation; remove it and the ceiling is no longer a function of your training data.

The 1M number is a design claim, not a measurement

That argument is elegant and it is also the whole basis for the 1M-token headline. The technical report publishes no RULER, no needle-in-a-haystack, no long-context suite of any kind. The window is argued from architecture, not demonstrated on a benchmark. Meanwhile the one public deployment write-up we could find — Backend.AI, serving the quantized model across two DGX Spark systems — set max-model-len to 200,000, not 1,048,576. The architectural ceiling may well be real. What quality survives at 500K or 1M tokens is simply not in the paper.

The efficiency claim carries the same asterisk: the report states the hybrid stack needs roughly a quarter of the memory and compute of an all-softmax stack at 1M context, and separately that the hybrid design reached target loss with about 3.2× fewer training tokens than all-softmax on MMLU. Both are Upstage's own figures. No throughput or latency numbers are published at all.

The benchmark card, read honestly

On the English suite, against DeepSeek V4 Flash, Solar Open 2 250B posts 86.2 on MMLU-Pro versus 85.9 and 92.4 on LiveCodeBench v6 versus 92.3 — differences small enough to be scoring noise. Then it posts 16.6 on APEX-Agents against 13.2, which is the one real gap and, unsurprisingly, the number Upstage leads with.

The losses are in the same report and get less airtime. SWE-Bench Verified: 70.4 against 73.8. GDPval-AA v2, on ELO: 1128 against 1187. Terminal Bench Hard: 28.3, against 41.7 for MiMo-V2.5. ArtifactsBench: 55.9, behind both frontier baselines. To Upstage's credit the report names these gaps and where they point — repository- and terminal-level software engineering, and numerical precision in office deliverables — rather than burying them.

The comparison that actually impresses is internal. Against Solar Open 1: LiveCodeBench v6 goes 56.5 to 92.4, SWE-Bench Verified 15.4 to 70.4, APEX-Agents 2.4 to 16.6. Whatever you make of the head-to-heads, that is a genuine generational jump in six months.

On Korean, the average across the suite is 85.4 against DeepSeek V4 Flash's 84.9 — but the average is doing work. Solar Open 2 250B loses KMMLU-Pro (78.4 to 78.9) and Ko-AIME'25 (97.7 to 98.0), and wins Ko-GDPval (86.8 to 85.0). The lead comes from the deliverable benchmark, not from knowledge or math.

Which brings us to the line that travelled furthest: that Solar Open 2 250B matches a model more than six times its size. The basis is Ko-GDPval, where it scores 86.8 against DeepSeek-V4-Pro's 86.9. That is one Korean benchmark, and it is a tie rather than a match-up win — while on the English suite, V4-Pro's smaller sibling already beats Solar Open 2 250B on SWE-Bench Verified. One benchmark does not generalise into a size-efficiency law. A related caution: baseline figures for APEX-Agents differ slightly between the paper and secondary coverage (13.2 versus 13.4 for the same models), so treat the ordering as directional rather than precise.

On preference, a blind pairwise test over 835 Korean conversations put Solar Open 2 250B ahead 44.1% to 26.6%, with the report attributing its alignment-and-honesty win to cultural grounding rather than extra safety training. Worth noting what is absent: the report contains no quantitative harm, refusal or jailbreak evaluation of any kind.

The Hugging Face card has since picked up a few more numbers in its evaluation-results section — GPQA Diamond 86.3, HLE 28.8, and MathArena AIME 2026 95.7 — but they were extracted from the model card itself rather than from an independent run, so they carry the same label as everything above: Upstage's figures, unreproduced.

One external number has now surfaced that does not come from Upstage's lab. On August 13, Korean financial media — Edaily, Seoul Economic Daily, ITDaily and iNews24, all publishing the same figures — reported the second-round evaluation scores of the government's sovereign-AI programme, measured on Artificial Analysis's Intelligence Index (AAII v4.1.1): Motif 3 at 47, Solar Open 2 250B at 37, SK Telecom's A.X-K2 at 35, and LG AI Research's K-EXAONE 2.0 0803 at 31. The 37 puts Solar Open 2 250B second of the four Korean finalists, ten points behind Motif 3. Label it precisely: this is press-reported — the score has not appeared on Artificial Analysis's own leaderboard, which still returns nothing for this model — and the same coverage notes that the AAII carries no Korean-language benchmarks, a real blind spot for a model whose domestic results are central to its pitch. But it is the first read on Solar Open 2 250B that does not trace to Upstage, and it is consistent with the rest of this article: strong agent and deliverable scores, with a lot of the rest still unproven by anyone outside the vendor.

Why nobody can serve it on stock software

The model card tells the story better than any commentary. Downloads last month now sit past 26,000, alongside 743 likes, sixteen community quantizations and three finetunes — and, in the Inference Providers box, the line "This model isn't deployed by any Inference Provider" still holds, with a dozen people now listed as asking the card for provider support. Real interest, zero hosted endpoints.

Follow the working deployments and they all lead to a fork. Nota AI's NVFP4 build pins its install to git+https://github.com/UpstageAI/vllm.git at tag v0.22.0-solar-open2 — Upstage's own vLLM. Backend.AI's DGX Spark write-up needed that fork plus a patched FlashInfer, because upstream FlashInfer had no expert-parallel support and would not take the compressed-tensors format.

Mainline support is pull request 51129 against vllm-project/vllm, opened on August 5 by a contributor going by oesni. It is still open: on August 10 mergify flagged merge conflicts, so the PR now carries a needs-rebase label and still awaits review. What that PR had to build is the most honest description of this model's difficulty that exists in public:

• A Solar-specific KimiGatedDeltaAttention layer in a new solar_open2.py, carrying the 2·sigmoid(b) beta from the negative-eigenvalue change

• The operation vllm::solar_open2_kda_attention registered as a CUDA-graph splitting op

• A profile-time KDA prefill warmup, capped at 2048 tokens, added specifically because real prefills were running about 23% slower without it

• Reasoning-output and tool-call parsers for both streaming and non-streaming modes

• Serving validation on 8×B200 reporting 107.02 GiB per rank of KV cache, alongside 463 passing unit tests

Those are the PR author's figures, unreproduced by anyone else, and the code can still change before it merges. But read that list next to the architecture section and the shape of the problem is obvious: the two choices that make Solar Open 2 250B cheap to run at long context — KDA in three quarters of the layers, and no RoPE — are precisely the choices for which general-purpose inference stacks have no existing code path. Openness of weights and openness of the path to running them are different things, and only one of them shipped on July 22.

What it costs to stand up

Start with the memory floor. 500.6 GB of bfloat16 weights against an H200's 141 GB means four GPUs give you 564 GB — about 63 GB left across the whole node for KV cache and activations before a single request lands. That is why the card lists four H200s as the minimum and recommends eight, and why FlexAI offers it as a dedicated 8×H100 endpoint with pricing on request rather than on a shared per-token API.

Quantization moves that floor a long way. Nota AI's NVFP4 build (4-bit weights and activations, group size 16, in compressed-tensors format) takes the model from 500.6 GB to 153.3 GB, and their own evaluation puts the accuracy cost at 0.22 points averaged over four benchmarks — 81.57 to 81.35, with Tau2-Bench 75.20 to 75.08, GPQA-Diamond 86.26 to 85.45, MMLU-Pro 86.19 to 86.15, and LiveCodeBench moving the other way, 87.03 to 88.55, which is quantization noise rather than an improvement. The catch is hardware: NVFP4 needs the FP4 tensor cores introduced with Blackwell, so B200 or GB200 only, nothing on Hopper. If you are on H100s, the variant you want is their INT4 globally-pruned build — 117.8 GB after pruning, sized to fit two of them.

Now the arithmetic that decides it. DeepSeek V4 Flash — again, the model Upstage chose to be measured against — runs $0.15 per million input tokens and $0.29 per million output on OrcaRouter, which is DeepSeek's list price passed straight through at 0% markup. At a 3:1 input-to-output mix that blends to roughly $0.185 per million. Put an illustrative $2 per GPU-hour on an eight-GPU node and you are at about $11,700 a month before anyone sends a request — equivalent to roughly 63 billion blended tokens of API traffic, or 2.1 billion tokens a day, every day, just to reach break-even. Quantize down to a two-GPU node and break-even falls to something like 16 billion tokens a month. Substitute your own GPU rate; the shape does not move. Self-hosting a 250B model is a decision about utilization and control, not a discount.

To be exact about what is and is not available: Solar Open 2 250B is not on OrcaRouter, and by its own model card it is not on any inference provider. What is on OrcaRouter is the field it is being measured against — DeepSeek V4 Flash, Qwen3.8-Max and a couple of hundred others behind one key at provider list price, with failover across providers if one goes down. Which matters here for a narrow, practical reason: if the question you actually want answered is "does an agent-tuned model beat what we already call in production", you can run that half of the comparison by changing a model string this afternoon, while the GPU quote for the other half is still in procurement.

Two strings on the license

The Upstage Solar License permits commercial use, fine-tuning and distillation, and Upstage describes it as Apache-2.0-based with brand terms added. The two added obligations are the ones to read carefully: a derivative model must carry the "Solar" prefix in its name, and you must display "Built with Solar" attribution.

For internal deployment that is a non-event. If you planned to distill this into something you ship under your own product brand, the naming clause is a branding decision dressed as a licence term, and it is worth a lawyer's twenty minutes before you start a training run. Read the clause on the repo rather than trusting coverage of it — reporting on this licence has been loose, including at least one write-up that describes it flatly as Apache 2.0.

Who should move now, and who should wait

Move if Korean or Japanese deliverable quality is your product and you can put the weights on hardware you already own — that combination is exactly what "open weights, no endpoint" is good for, and Upstage's own framing is explicit that sovereignty means being able to deploy and verify a model yourself, not just that it was trained domestically. Move if you have idle GPUs and an agent workload with enough volume to fill them, because at real utilization the per-token maths inverts. Move if audit or data-residency requirements mean an API was never on the table.

Wait if you want a per-token endpoint. Upstage has said it intends to list Solar Open 2 250B on a third-party platform; as of writing that has not happened — the Playground trial ran only to July 31, and the model card still shows no provider. Wait if you are on Hopper and were counting on the 4-bit build, since the NVFP4 version needs Blackwell. And wait before designing anything around the 1M context number, which is currently a property of the architecture rather than a measured result.

The thing to watch is pull request 51129, which on August 10 picked up a needs-rebase label after mergify flagged merge conflicts. The day Solar Open 2 250B appears in a tagged vLLM release is the day it stops being an infrastructure project and becomes a model anyone can simply try. The press-reported AAII of 37 is a start — a first external read, however provisional, and one that still does not appear on Artificial Analysis's own leaderboard. The real test is the one that runs on stock software, and it has no date yet.