ARTEMIS vs Qwen3.8: the same benchmark, two different jobs

Google's ARTEMIS and Qwen3.8 are both measured on AndroidWorld, and that coincidence is the single most misleading fact in either project's launch coverage. ARTEMIS is an Android automation harness — open-sourced by Google in August 2026 under Apache 2.0, it takes a natural-language instruction, drives a real handset through ADB, and claims a 99%+ completion rate on Google Research's 116-task AndroidWorld benchmark, showing 99.1% on the public leaderboard as of September 11, 2026. Qwen3.8-Flash is Alibaba's multimodal mixture-of-experts model, released and open-sourced on August 26, 2026, whose launch material claims it beats Claude Opus 4.6 by 22.5 points on AndroidWorld. One of those numbers is a system's score. The other is a model's contribution to a system somebody else wrote. Read them as rivals and you will conclude the wrong thing about both; read them as two halves of one stack and the interesting question — what a long Android run actually costs — finally has an answer.

What Qwen3.8 is, per size

"Qwen3.8" is a family, not a checkpoint, and the member that matters here is not the flagship.

• Qwen3.8-Max — the flagship tier, positioned against frontier hosted models.

• Qwen3.8-27B — the open mid-size dense sibling.

• Qwen3.8-Flash — a multimodal MoE on a new "Next" architecture: 125B transformer parameters with only 6B activated per token, Qwen Sparse Attention fused with GDN in a hybrid attention scheme for long-context speedups when the cache hits, a Gated Residual splitting the information channel four ways, and 51B parameters of N-gram embeddings. Alibaba describes the Next architecture as the prototype for the next-generation Qwen4.

• Context — natively large, with most listings at 1M tokens and some sources describing 262K native extended to 1M. Max output is around 131K.

• Price — the published API rate is ¥1 per million input tokens and ¥3 per million output, which lands at $0.15 / $0.47 on our catalogue, with cache reads at $0.018 and cache writes at $0.230. Alibaba's own framing is that cache-hit pricing as low as ¥0.1 per million is what makes long-input agent workflows viable, and the numbers support that: a cache read costs roughly a twelfth of a fresh input token.

• The open-weights question — the Flash weights were published on Hugging Face and ModelScope on release day. Note how the family is counted: Alibaba says the Qwen3.8 series has open-sourced three sizes — Max, 27B and Flash — for 2.4T parameters in total, which is a cumulative figure across the series rather than any one model's parameter count.

The benchmark claims are broad and, per Alibaba's own launch material, all deltas rather than absolutes: SWE-bench Pro +9.1 over Opus 4.6, JobBench around +20, MathVision +25.1, ERQA +31.5, AndroidWorld +22.5. Deltas against an unnamed configuration of a competitor model are not the same category of evidence as a leaderboard entry, and none of these have been independently reproduced. The company's headline framing — a redefinition of the industry's cost "kill line" from price-per-token to total cost per completed task — is the claim that actually matters for this comparison, and it is also the one you can test yourself.

What ARTEMIS contributes, and why the two numbers are not comparable

ARTEMIS contains no model. Its badge reads "Multi-Model — Gemini | Claude | GPT-4o | Qwen-VL" and its configuration lives in code>config/artemis.jsonc/code>. What it brings is the part that turns a model's guess into a verified action: a dynamic-first locator that reads the accessibility tree when it can and falls back to vision and coordinates when a Compose or Flutter surface gives it nothing, a Safety Net that clears interfering system popups before the tap lands, checkpoint verification at four levels from code>off/code> to code>strict/code>, and a session ledger that compresses old screenshots into visual summaries so a hundred-step context stays affordable.

It also ships a native MCP server. code>uv run artemis mcp --install all/code> exposes code>mobile_run_task/code>, code>mobile_manage_task/code>, code>mobile_get_device_state/code>, code>mobile_inspect_trace/code> and code>mobile_diagnose/code> to Antigravity, Claude Code, Codex and anything else that speaks MCP — which is what made the project travel as fast as it did, and which is the cleanest statement of what it is for. ARTEMIS is a tool an agent calls. So is a model, which is why putting them in the same column is a category error.

The one thing to be careful about when reading ARTEMIS's 99.1%: AndroidWorld's leaderboard explicitly does not independently verify submissions. Every figure on it, ARTEMIS's included, is self-reported by the team that produced it. The same caveat applies with more force to a delta quoted in a launch post.

Reading "vs" two ways

There are two honest readings of this matchup and they point in opposite directions.

As rivals, the comparison is really: "should I use a hosted model plus a harness, or should I use a model that is good enough at mobile tasks that I can write the harness myself?" On that reading ARTEMIS wins by default, because a model is not a harness — and the +22.5-point AndroidWorld delta does not tell you whether Qwen3.8-Flash can survive step 74 of a hundred-step run when a system popup eats its tap. Nothing in a benchmark table measures the Safety Net.

As a stack, the comparison is the useful one: ARTEMIS is the control loop, Qwen3.8-Flash is a plausible engine for it, and the question becomes one of cost and reliability at length. Alibaba's whole pitch for the Flash tier — that the important number is total cost per completed task rather than price per token — is precisely the metric an Android automation suite is scored on, and it is a metric you can measure on your own test set in a day.

The awkward detail standing in the way: Qwen3.8-Flash is not on ARTEMIS's tested backend list, which names Gemini, Claude, GPT-4o and Qwen-VL. Qwen-VL is a different model. The harness takes a model endpoint and the pairing is technically plausible, but nobody has published an evaluation of it, and if you run it you are doing original work rather than following documentation.

The arithmetic that decides it

Here is the calculation worth doing before either launch post persuades you of anything. It is a model with stated assumptions, not a measurement, and you should substitute your own numbers — but the shape of it is the point.

Take a 100-step Pro run. Pro runs at roughly 15–40 seconds per step, so the wall clock is somewhere between 25 minutes and an hour, and every step sends a screenshot plus a growing prompt. Assume 8,000 prompt tokens and 300 output tokens per step — a conservative screenshot-and-instructions budget, well under what a raw full-resolution frame costs. That is 800,000 input tokens and 30,000 output tokens for one test.

• At Qwen3.8-Flash's $0.15 / $0.47, that run costs about $0.12 in input and $0.014 in output — roughly thirteen cents.

• At a frontier hosted rate in the low single digits per million input and mid-teens per million output, the same run lands in the dollars, not cents. The two rates are hand-waved here on purpose, because the exact figure depends on which frontier model you pick and the ratio is what matters: it is an order of magnitude, on every test, every run.

• And because ARTEMIS re-reads a growing context on every step, the ratio improves further for whichever model has the cheaper cache read. Qwen3.8-Flash's cache read is $0.018 per million against a $0.15 fresh input — a twelfth of the price, and the reason Alibaba keeps pointing at cache-hit rates when it talks about agent workloads.

Now multiply by your suite. A 500-test regression run nightly is 400 million input tokens a night, and the difference between thirteen cents and three dollars per test is the difference between a harness you can afford to run continuously and one you schedule weekly.

Running it, and where the plumbing matters

If you want to test that arithmetic on your own app, the honest route is to point ARTEMIS at a model endpoint and measure. Qwen3.8-Flash is on our catalogue at the published $0.15 / $0.47 with cache reads at $0.018 and cache writes at $0.230, on the same key and the same bill as the other Qwen3.8 sizes and everything else we route — which is the part that matters when the workflow you are pricing is a harness making a hundred calls per test rather than a person making one. The model page also carries the operational numbers you want before you commit a suite to it: a 1M-token context with 131K maximum output, a p50 time to first token of 7.12 seconds and a p95 of 10.00, and roughly 2.3 billion tokens of traffic through it in the last seven days. That last figure is ours rather than a vendor's, and it is a reasonable proxy for whether other people are already running long agent workloads on this endpoint.

The plumbing detail that stops being theoretical at this scale is what happens at step 74. A Pro run holds its context on one endpoint for the better part of an hour; a provider incident in the middle of it does not degrade the run, it ends it, and you pay for the 73 steps that did not produce a result. Routing a long agent session through a layer that fails over to another provider of the same model is the difference between a flaky suite and an interrupted one. That is a mundane engineering property, and it is the one that decides whether your measured cost per completed task survives contact with a week of real runs.

What each one is actually for

If you have an Android app and a test suite, you want ARTEMIS, and Qwen3.8-Flash is a candidate engine for it rather than an alternative to it — an undocumented one, worth an afternoon's experiment, priced so that the experiment costs you nothing measurable. If you are choosing a model for a mobile agent you are writing yourself, the +22.5-point AndroidWorld delta is a reason to look at Qwen3.8-Flash and not a reason to believe it, because it is a vendor-reported delta with no absolute score attached and no independent reproduction.

The thing both projects are quietly arguing about is the same thing, and neither says it plainly. Google is claiming that the harness is what makes a mobile agent reliable, and open-sourcing it so the claim can be checked. Alibaba is claiming that cost per completed task is the only number that matters, and pricing accordingly. Both are probably right, which is why the interesting configuration is not ARTEMIS or Qwen3.8 — it is ARTEMIS on Qwen3.8-Flash, measured on your own app, with the trace left switched on.

And because ARTEMIS re-reads a growing context on every step, the ratio improves further for whichever model has the cheaper cache read.