GPT-6.1 Sol on COBOL: 86.6% Against 5.4%, and Only One of the Two Tests Can Be Rerun

On October 19, 2026, O​penAI published two coding evaluations for low-resource languages, and the headline number belongs to GPT-6.1 Sol: 86.6% of COBOL migration tasks passed at low reasoning effort, against 5.4% for GPT-5.6 Sol at the same setting, rising to 97.9% when effort goes up. The second evaluation, MultiLangBench, is a different model's story — GPT-6 Astra averaged 87.4% across twenty languages at maximum effort — but it is the more useful half of the release, because O​penAI put its harness on GitHub under Apache-2.0 and you can run it yourself this afternoon. The COBOL migration eval, the one carrying the 86.6%, is a different matter: its tests are hidden, its pass criterion is byte-for-byte output parity against cases nobody outside O​penAI has seen, and no third party has reproduced it. That asymmetry is the whole story, and it is worth stating plainly before either number gets quoted in a migration business case. The documented AI failures and agent incidents behind releases like this one are collected in the OrcaRouter AI Incident Archive where each entry carries its own sources, severity, confidence grade and disputed flag.

One date to keep straight. GPT-6.1 Sol itself is from September 29, 2026 — this is not a launch, and nothing here changes the model's weights, price or rate card. What happened on October 19 is that O​penAI published evidence about what the model can do in a category it had not previously documented, and released a benchmark anyone can rerun. That is the event this piece is about.

What was actually published, and what each half is for

The two evaluations answer different questions, and O​penAI's own framing separates them. MultiLangBench asks whether a model can build something in a language nobody trains on: implement SQLite 3.40.1 from scratch in one of twenty programming languages, then pass a database conformance suite. The COBOL migration evaluation asks whether a model can move something: translate COBOL programs into C++, Java, Kotlin, Python or Rust while preserving observable behaviour exactly.

The first is a capability floor test. The second is the one enterprises actually care about, because COBOL is not a benchmark curiosity — IBM's long-standing estimates put COBOL behind more than 40% of online banking systems and around 95% of ATM transactions, and the code is maintained by a workforce that is not being replaced.

• MultiLangBench, maximum effort — GPT-6 Astra 87.4%, GPT-6 Luna 40.3%, GPT-5.6 Sol 26.8%. Astra cleared 83% in 19 of 20 languages; COBOL was its weakest at 72.1% against GPT-5.6 Sol's 10.9%, which tells you the hardest language in the set is still the one with the oldest tooling.

• COBOL migration, low effort — GPT-6.1 Sol 86.6%, GPT-5.6 Sol 5.4%, at an estimated 61% of the older model's API cost for the evaluation run, or 39% less.

• COBOL migration, higher effort — GPT-6.1 Sol 97.9%, which O​penAI describes as close to saturation.

• Per-language detail O​penAI did publish — Fortran 83.5% for Astra against 21.4% for GPT-5.6 Sol; Visual Basic 89.4% against 27.5%; OCaml 85.8% against 25.6%; GPT-6 Luna at 17.1% on COBOL, 31.4% on Fortran, 34.1% on OCaml and 41.6% on Visual Basic.

• Cost per sample on MultiLangBench — an estimated $0.77 for GPT-6 Luna against $3.48 for GPT-5.6 Sol, roughly 78% lower, which is the number that makes the weakest model on the board the interesting one for a build-generation pipeline.

The eval you can rerun: what MultiLangBench actually asks for

The repository is github.com/openai/MultiLangBench, Apache-2.0, and reading the harness is more informative than reading the leaderboard. The task is specific enough that a model cannot bluff it: produce an executable that takes a database path as a command-line argument, reads an SQL query from stdin, and behaves like SQLite 3.40.1. The tests come from SQLite's own TCL suite — about 51,000 tests in total, of which roughly 24,000 were converted into the triplets the harness uses: an initial database, a user query, and a validation query. Test names are SHA-256 hashes of those components, so the suite is not memorisable by name.

The rules are where the difficulty lives, and each one closes a shortcut:

• No internet, and no SQLite anywhere in the environment. The harness says to strip every trace of SQLite first. A model cannot shell out to the real engine, and it cannot fetch an implementation.

• Validation runs against real SQLite 3.40.1. Your implementation modifies the database; the harness extracts it and checks it with the genuine article. Approximate behaviour fails.

• Sixty seconds per test, and a wrong-language answer scores zero. Mixing languages fails, and the harness notes that another model may be used to check which language you actually wrote in. That last rule exists because the obvious cheat — write it in Python and wrap it — is the first thing a capable model tries.

• The quick run is 272 tests, not 24,000. The README recommends running only the tests whose names begin with 00, 01 or 02 when you want a comparable number. That is the detail to note before you compare your own score to O​penAI's: a full-suite run and a quick run are not the same measurement, and the README says so.

That is a real, checkable benchmark. It also tells you something the press coverage did not: the 87.4% headline was measured on a suite you can point at your own model, which is rare enough to be worth the download.

The eval you cannot rerun, and why that is not automatically a scandal

The COBOL migration evaluation is O​penAI-designed, O​penAI-run and O​penAI-scored, and its test cases are not published. The pass bar is stated precisely — a migration counts only if every hidden test produces byte-for-byte identical observable output — and that precision is exactly what makes the result unverifiable from outside: the 86.6% is O​penAI's measurement against O​penAI's own hidden cases. Only one of the two evaluations in the release is something a reader can reproduce.

Withholding migration tests is not indefensible. The corpus is presumably drawn from real legacy programs, and publishing it can leak customer code or hand future models a training set. Byte-for-byte parity is also a genuinely conservative criterion, stricter than most vendor evals: it does not reward a translation that is merely close.

But the criterion is also where the practical trouble starts, and O​penAI's own caveats point at it. Migration output still needs expert review and testing against business rules and system dependencies before production. And byte parity is hard in exactly the places COBOL is hard: the language truncates and rounds fixed-point arithmetic in ways most target languages do not, so a translation can look correct, pass a suite that checks "close enough", and still move a balance by a cent. A hidden suite that requires exact equality handles that; your own review process has to handle it too, because the model's score is not a statement about your programs.

The cost figure needs the same care. The 39% saving is an estimate for the evaluation run — tokens in and out of the benchmark — not the cost of replacing a production system, which is dominated by discovery, expert review, integration testing and the years of undocumented behaviour that no eval samples.

What 86.6% and 97.9% should change in a decision

The honest reading is narrower than the headlines and more useful than them. A 5.4% baseline is not a model that half-works; it is a model that almost never produces a behaviour-preserving translation, so the older generation was never a candidate for this work. GPT-6.1 Sol at 86.6% is in the range where the economics invert: a tool that drafts most of a translation and leaves a reviewer to check it is a different product from one that produces nothing usable.

The 97.9% at higher effort is the number to be sceptical of, not because it is wrong but because it is saturated. Once a benchmark is near its ceiling it stops discriminating between models and starts measuring the suite, and O​penAI says as much. For planning, treat 86.6% at low effort as the cheap operating point and 97.9% as the ceiling of what the eval can show you — not as an accuracy promise for your codebase.

There is also a cross-check worth doing, and it cuts against the hype. In the build test, COBOL remained GPT-6 Astra's weakest language at 72.1% even though Astra is the stronger model overall at 87.4% average. The language is hard in both directions. A model that migrates COBOL well in a hidden eval is not thereby good at COBOL; it is good at the pattern the eval samples.

The other thing that happened this week: the reasoning-trace claim

A widely shared post on October 21 said GPT-6.1 Sol's reasoning traces are "totally legible, nothing to worry about." Nothing in O​penAI's October 19 publication, and nothing on the model's own documentation page, supports that as a change in what the API returns. O​penAI's reasoning guide is explicit that raw reasoning tokens are not exposed — reasoning items remain opaque — and that what you can request is a summary of the reasoning through the summary parameter, with encrypted reasoning content replayed across turns. GPT-6.1 Sol supports the all_turns form of that replay. A summary channel is not a legible trace, and treating the two as the same thing is how a safety claim gets made by screenshot.

What is measurable is adjacent and more interesting for anyone auditing an agent. Sonar's October 9 evaluation of GPT-6.1 Sol, run against GPT-6 Sol at medium effort over 4,444 Java tasks, found reasoning tokens fell 51.4% — from 2.64M to 1.28M — while reasoning's share of output dropped from 41.8% to 19.5%, output rose, and the functional pass rate improved from 83.09% to 85.29%. The same run found vulnerability density down 32.4% but critical-severity reliability findings up from 21 to 46, with concurrency bugs the largest category at 196 findings. That is a trade a reviewer can act on. "Traces are legible" is not, because the traces are not what is being returned.

For a migration project this matters in one concrete way: your audit trail is the diff and the test output, not the model's narration. Budget review capacity accordingly — Sonar's finding that 39.4% more generated code produced 12,982 total findings against 9,859 is the shape of the cost that hides behind a benchmark score.

Running both arms yourself

The practical problem with a comparison like this is that the baseline arm and the new arm are usually two contracts, two keys and two billing relationships, which is enough friction that nobody runs the baseline. Both models sit on our catalogue behind one key: openai/gpt-6.1-sol and openai/gpt-5.6-sol, at O​penAI's list prices with 0% markup, so the same $2.00 input and $10.00 output per million tokens you would pay the vendor, with cached input at $0.10 and cache writes at $2.50, and the above-272K tier passed through at $4.00, $0.20, $5.00 and $15.00. A vendor rate change is live here the day it is published rather than at the next contract review.

That pass-through is what makes a same-day benchmark cheap to run: point the MultiLangBench quick suite at both models, pay list price on each, and you have a number about your own toolchain rather than O​penAI's. Automatic failover is the other half of the argument — if the newer model degrades mid-run or a route gets rate limited, the request does not fail with it, which is what lets you put an unproven model in front of a migration pipeline before you have your own evidence.

The MultiLangBench harness is public and the COBOL migration cases are not, so the only honest test of the 86.6% is the one you build: sample a few hundred of your own COBOL programs, translate them, and diff the output byte for byte against the original behaviour. That is a week of work and it is worth more than the vendor table.

What to watch next

Three things would change this picture, and none has happened yet. O​penAI publishing the COBOL migration cases, or an independent lab re-running them, would turn a vendor number into a measurement. A third party reproducing the MultiLangBench per-language table would tell us whether COBOL's 72.1% is a property of the language or of the suite. And the model's independent benchmark record — Artificial Analysis currently puts GPT-6.1 Sol at an Intelligence Index of 52, twelfth of 214 models, at 55.2 output tokens per second and $0.72 per index task — is the check that does not depend on O​penAI's own evals at all.

Until then the useful summary is short. GPT-6.1 Sol is a model from September 29 whose legacy-code capability was documented on October 19, in one eval you can run and one you cannot. The runnable one says the new generation is in a different class on low-resource languages. The unrunnable one says COBOL migration is now worth a pilot. Only your own codebase can turn either into a decision.