ARTEMIS vs Gemma 4 12B: a harness and a brain are not the same purchase

Google's ARTEMIS and Gemma 4 12B do not compete, and the fastest way to waste a week is to put them in a comparison table and pick a winner. ARTEMIS, open-sourced by Google in August 2026 under Apache 2.0, is an Android automation harness: it takes a sentence, drives a real phone through ADB, and reports what happened. Gemma 4 12B, released by Google DeepMind on June 3, 2026, is a dense 12-billion-parameter open-weights multimodal model — a brain you can run on a laptop, not a system that can tap a screen. One of them cannot see, decide or act without the other kind of thing attached; the other cannot hold a device at all. But the comparison is worth making, because the question underneath it is the one every mobile team is currently asking: can a small open model you own outright do Android automation, or do you need a hosted frontier model behind a harness like ARTEMIS? That question has a real answer, and it is not the one either vendor's launch post implies.

First, the category error, stated plainly

ARTEMIS contains no model. Its own badge reads "Multi-Model — Gemini | Claude | GPT-4o | Qwen-VL", and the file that decides which one you use is code>config/artemis.jsonc/code>. It is a control loop: an accessibility-first locator, a safety check that clears popups before your tap lands, a session ledger that compresses old screenshots into visual summaries, and two execution profiles — Flash at roughly 3–5 seconds per step, Pro at roughly 15–40 seconds per step with a Planner, an Operator and a read-only Checker. Everything it knows about the world arrives through a vision-language model it did not write.

Gemma 4 12B is the opposite shape of object. It is a checkpoint. It has no device connection, no ADB, no accessibility service, no notion that a tap is a thing that can be executed. Give it a screenshot and ask what to do next and it will answer you — possibly well — but the answering is the entire extent of its agency. Everything between "the model said tap at (540, 1180)" and "the phone tapped at (540, 1180)" is ARTEMIS's job, and it is most of the work.

So the honest framing of this matchup is not ARTEMIS-versus-Gemma. It is: what does a 12B open model buy you as the reasoning layer of an Android agent, and when do you need to pay for something larger?

What Gemma 4 12B actually brings

For a mid-sized model it is unusually well specified, and three of its properties matter for anything mobile.

• It is genuinely multimodal at the input layer. Text, image, audio and video go in; text comes out. It is the first mid-size Gemma with no separate multimodal encoders, and the first in the family to ingest audio natively — which is not a UI-automation feature, but is a real one if your test scenarios involve voice notes, notifications that speak, or accessibility paths built on audio cues.

• It is a 12B you can hold. Artificial Analysis lists it at 12B total parameters under Apache 2.0 — no revenue gate, no research clause, a licence you can build a product on without asking anyone — with a 256K context window, reasoning enabled, and a measured output speed of 109.2 tokens per second. Community reports put the weights at roughly 13.4 GB at SFP8 and about 6.7 GB at Q4_0, which is a laptop with 16 GB of memory.

• It is cheap, but not the cheapest thing in its class. Artificial Analysis lists it at $0.10 per million input tokens and $0.30 per million output — and notes in the same panel that this is on the expensive side for an open-weights model of similar size, where the median sits at $0.05 in and $0.15 out. Cache reads run near $0.03.

The benchmark picture is the usual mid-size muddle and worth stating carefully, because the numbers on the tracker sites disagree with each other. Artificial Analysis scores the reasoning configuration at an Intelligence Index of 14, ranking it 21st of the 142 open-weights models in its class — which is a respectable mid-table position and a long way from the frontier. Google's own positioning is that the 12B nearly matches the larger Gemma 4 26B-A4B and beats the previous generation's Gemma 3 27B on reasoning, science and document tasks. Third-party aggregators put it at roughly 72% on LiveCodeBench v6 and 77.2% on MMLU-Pro, with GPQA Diamond ranging from 66% to 79% depending on which tracker and which configuration you read. Those are respectable numbers for a 12B. They are also unaudited, self-reported aggregates, and where four sites disagree by thirteen points you should hold the range rather than the point.

What ARTEMIS brings, in one paragraph, because it is not the subject here

ARTEMIS brings everything the model cannot: a dynamic-first locator that uses accessibility element indices when they exist and falls back to vision and coordinates when they do not, which means no XPath to maintain and no IDs to go stale on a Compose or Flutter surface. It brings a Safety Net that intercepts the system popup your tap was about to land on, checkpoint verification with four levels from code>off/code> to code>strict/code>, automatic crash stacks, keyframe screenshots and diagnostic reports, a web console, a Python SDK for pytest and CI, and — the reason it spread as fast as it did — a native MCP server that exposes the whole thing to Antigravity, Claude Code, Codex and any other MCP-capable assistant as five tools. Its claimed 99%+ completion rate on Google Research's AndroidWorld benchmark puts it at 99.1% on the public leaderboard as of September 11, 2026, against 80% for human performance. That figure is self-reported and the leaderboard does not verify submissions, so treat it as a strong vendor claim rather than an audit.

The one place the two genuinely overlap

There is a real architecture in which a small Gemma does the whole job, and it is worth looking at because it shows what the cloud model is actually paying for. An open-source Android agent framework called Orion runs entirely on-device: MediaProjection captures the screen, a quantized Gemma-4-E4B-it running on the Qualcomm Hexagon NPU via LiteRT-LM chooses the next action, and the Android Accessibility Service dispatches the taps. No cloud API, no per-token bill, and the screen never leaves the handset. It is validated on a Galaxy S25 Ultra and, by its own description, is meaningful only on hardware with a Hexagon NPU — the model needs about 3 GB of storage and the framework targets QNN HTP v79 on arm64.

That is the shape of a small-model Android agent working today, and note what it costs to get there. The model is quantized and NPU-mapped. The action space is pixels, because a screenshot-only model cannot interpret "element index 12". The whole design is a vertical stack — model, runtime, silicon, framework — which is why it is a framework and not a drop-in for your test suite. Gemma 4 12B is roughly three times the parameter count of the E4B in that stack and is not shipped in a phone-runnable format; a 12B at four-bit quantization is a workstation or a served endpoint, not an NPU resident.

Where each actually wins

• Cost at scale — a self-hosted Gemma 4 12B has no per-token price at all, versus a per-step vision call for every step of an ARTEMIS run. On a 500-test regression suite run nightly, that difference compounds faster than any benchmark gap.

• Privacy — Gemma 4 12B on your own hardware never sends a screenshot anywhere; ARTEMIS's accessibility helper is explicitly on-device and sends nothing, but the model call it makes with each screenshot goes to whichever provider you configured.

• Task reliability on a real app — ARTEMIS, by a wide margin, because it is a harness with a safety net, checkpoint verification and recovery. Nothing about a better model substitutes for that.

• Audio and long documents — Gemma 4 12B, with native audio input on the technical specification sheet and a 256K-token context window. ARTEMIS has no opinion on either.

• Time to first passing test — ARTEMIS, by an enormous margin. Clone, code>./start.sh/code>, connect a device with USB debugging, wait for the first task to install the accessibility helper. Building the equivalent with a bare checkpoint is a multi-month project, and the Orion example above is what that project looks like when it is done.

• Freedom to modify the reasoning layer — Gemma 4 12B, which is Apache 2.0 weights you can fine-tune on your own UI vocabulary. ARTEMIS is Apache 2.0 code, but the reasoning is wherever your model lives.

The versions of this pairing that are actually available today

Be clear about what you can deploy this week. ARTEMIS's tested backend list is Gemini, Claude, GPT-4o and Qwen-VL — Gemma 4 12B does not appear on it, and there is no published evaluation of Gemma 4 12B driving an ARTEMIS session. The config file takes a model endpoint, so the pairing is plausible rather than proven, and if you try it you are doing original work rather than following documentation. Worth saying plainly: ARTEMIS's roadmap has an "on-device lightweight VLMs" item, and the model that fills it will not be a 12B.

Gemma 4 12B is not on our catalogue either — we route the other Gemma 4 sizes, Gemma 4 26B-A4B and Gemma 4 31B, and not the 12B, so if that is the checkpoint you want you are getting it from the vendor's own distribution and the usual third-party hosts. What a routed catalogue is for is the other half of this problem: if your plan is ARTEMIS on a hosted vision model, that model is a line item that scales with every step of every run, and the useful lever is not the sticker price but the cache price. A Pro run re-reads a growing context on every step, so a model with a cheap cache read changes the arithmetic far more than a model with a cheap fresh input. That is also why provider failover belongs in the config rather than in your incident log — a long Android session holds its context on one endpoint, and a provider hiccup at step 74 costs you the run.

Which one is your next move

If you have a mobile app and a regression suite, you want ARTEMIS, and Gemma 4 12B is not a substitute for any part of it — it is at best the engine you might swap in later, undocumented, on your own time. If you are building a product that must run on a handset with no network and no per-call cost, you want a small model, and Gemma 4 12B is probably too large for the form factor you actually have; look at what Orion did with a 4B-class Gemma on an NPU before you assume a 12B will fit.

The two decisions only collide in one place, and it is a cost question rather than a capability one: how many vision calls per day are you willing to buy? Answer that honestly — count your tests, count your steps, count your nightly runs — and the choice between a harness on a hosted model and a self-hosted stack makes itself.

What a routed catalogue is for is the other half of this problem: if your plan is ARTEMIS on a hosted vision model, that model is a line item that scales with every step of every run