Google's ARTEMIS open-sources Android automation: what the 99% AndroidWorld claim actually covers

The newest commit on google/artemis is not a feature. It is a credit line — "fix: complete README and relevant file headers per Apache 2.0 requirements," pushed September 12, 2026, three days after Minitap published a post titled I expected better from Google. Google's ARTEMIS is the company's newly open-sourced Android automation agent: it turns a plain-English instruction into real taps, swipes, typing and verification on a physical Android device or an emulator, captures the logs and screenshots along the way, and reports a 99%+ task completion rate on Google Research's AndroidWorld benchmark. It was published this August by Google's Pixel Test Engineering Fusion team under Apache 2.0, and it is genuinely worth your afternoon. It is also, as of the middle of September, the centre of an open-source attribution fight that says more about how mobile agents get built than the benchmark number does. Both stories are real. Only one of them is why the repository's last commit was a licence fix.

What actually shipped

ARTEMIS is not a model and not a chatbot wrapper. It is a control harness — a Python 3.12+ system that lives between a vision-language model and a real handset. You give it a task in English; it observes the screen, decides an action, executes it through ADB, checks what happened, and keeps going. Five things came in the box:

• A CLI and a Python SDK. code>./start.sh/code> bootstraps ADB, scrcpy, FFmpeg and the uv environment; code>uv run artemis run "…" --profile flash/code> runs a task headlessly; the code>artemis-client/code> SDK wraps the same call for pytest and CI, returning code>succeeded/code>, code>status/code>, code>device_serial/code> and a code>trace_id/code> you can chase later.

• A web console. code>uv run artemis ui/code> serves a visual test console on code>localhost:8000/code> where you watch the loop step by step.

• A native MCP server. This is the part that made it travel. code>uv run artemis mcp --install all/code> exposes code>mobile_run_task/code>, code>mobile_manage_task/code>, code>mobile_get_device_state/code>, code>mobile_inspect_trace/code> and code>mobile_diagnose/code> to any MCP-capable assistant, with first-class install paths for Antigravity, Claude Code and Codex, plus config generation for Cursor, Windsurf, VS Code and Cline/Roo. From an assistant that has the server connected, "build the APK, install it, open settings, toggle airplane mode, screenshot the result" stops being a script and becomes a sentence.

• An accessibility helper. The first task installs the Artemis Accessibility Helper, which reads the screen layout without holding the UiAutomation connection — deliberately, so that other UiAutomator-based tooling on the same device is not suppressed. It runs on the phone and sends nothing off it, and falls back to UIAutomator2 when it cannot attach, with the fallback surfaced in the task timeline.

• Log and trace capture. Crash stacks, keyframe screenshots and a diagnostic report are collected automatically rather than bolted on by the caller — the reason it is usable as a regression suite and not just a demo.

Flash and Pro are two different products sharing one name

The single most important thing to understand before you benchmark ARTEMIS against anything is that code>--profile flash/code> and code>--profile pro/code> are not a speed setting on one agent. They are two agents.

• Flash — a reactive observe-and-act loop at roughly 3–5 seconds per step, no plan, no notes, no pre-execution safety check, no checkpoint verification, no final report and no ADB shell. The loop is unbounded by default because history is compressed rather than accumulated.

• Pro — a multi-agent graph at roughly 15–40 seconds per step, built from a Planner that maintains a living Markdown plan with explicit code>verify/code> and code>assert/code> items, an Operator with the full toolset, and a read-only Checker that validates checkpoints and runs an exit review. code>--verification-level/code> takes code>off/code>, code>final/code> (the default), code>checkpoints/code> or code>strict/code>.

That is a five-to-ten-fold latency difference for the same task description, and it is the difference between a smoke test and a 100+ step exploratory run. Google's own framing is that Pro is for long-horizon work and continuous code>[Loop:continuous]/code> monitoring; Flash is for routine, deterministic UI chores. If you read a review that quotes step timings without telling you which profile produced them, it is not telling you anything.

The locating strategy is the real engineering

Most mobile automation frameworks die on selectors. ARTEMIS is built dynamic-first: when an accessibility element index exists, it uses it; when it does not — a Canvas, a Compose surface, a Flutter view, a game — it falls back to coordinates and vision. There is no XPath layer to maintain and no ID to go stale, which matters because the applications you most want to test are the ones that ship new builds weekly.

The Pro profile adds a Safety Net: every action passes a pre-execution check, XML-first with a pixel fallback, that catches the system popup about to eat your tap. Failures open an "execution incident" that stays in context until a later success resolves it, rather than spawning a separate repair agent. Long sessions get compressed — old screenshots become visual summaries, completed steps get chunked into recallable eras that code>search_history/code> and code>replay_steps/code> can pull back — which is what keeps a 100-step context from becoming unaffordable.

99.1% on the leaderboard, and the caveat the launch posts skip

ARTEMIS's headline claim is a 99%+ completion rate on AndroidWorld, Google Research's benchmark of 116 realistic tasks across 20-odd apps, scored as Pass@1. As of September 11, 2026 the AndroidWorld leaderboard listed ARTEMIS at 99.1% against Minitap's mobile-use at 91.4%, with human performance at 80%. On the table, that is state of the art among publicly reported results.

Two things belong next to that number. First, the AndroidWorld leaderboard explicitly does not independently verify submissions — every figure on it, including ARTEMIS's, is self-reported by the team that produced it, and a robustness analysis has shown that task variations alone can move an agent's score substantially. Treat 99.1% as a strong vendor claim on a public, checkable benchmark, which is a real and useful thing, and not as an audited measurement, which it is not. Second, the shape of the comparison matters: the benchmark is a fixed task set, and ARTEMIS's published comparison chart reportedly omitted mobile-use while including other entries. A benchmark where the strongest prior result is missing from the chart is a weaker claim than the raw percentage suggests.

The dispute, which is the actual news this month

On its blog and in a public issue on the repository, Minitap — a mobile-testing startup whose open-source mobile-use project does the same job for Android and iOS — alleged that 228 of ARTEMIS's 229 files were identical to its own. The specifics it published are unusually concrete: Android device-connection code matching its implementation, the word-for-word reuse of instructions belonging to an agent named "Hopper", and a WhatsApp example that sends New Year messages to Alice, Bob and Charlie, reproduced with the same comments, the same cleanup steps and the same bug. It further alleges that a file carrying the names of three Minitap authors — Pierre-Louis Favreau, Jean-Pierre Lo and Nicolas Dehandschoewercker — was replaced by force-push in August, with the names removed and a different author substituted.

The licence question is not murky. mobile-use is Apache 2.0, and Apache 2.0 permits exactly this kind of reuse — commercial, derivative, closed — provided you preserve the copyright notices and state what you changed. What it does not permit is shipping the code with the notices stripped. Since the allegations surfaced, the repository has carried the line "This project includes source code developed by Minitap, Inc." and links to minitap-ai/mobile-use, and the September 12 commit completing that attribution is, at the time of writing, the most recent change on code>main/code>. Minitap has published no evidence linking the attribution removal to its own unanswered leaderboard submissions, and Google has not issued a detailed public response. The honest reading: the code being shared is legitimate and was always permitted; the paperwork was, for a period, not, and it has now been corrected.

The part nobody costs out: the model bill

ARTEMIS ships with a badge reading "Multi-Model — Gemini | Claude | GPT-4o | Qwen-VL", and its configuration file at code>config/artemis.jsonc/code> is where you point it at whatever vision model you hold credentials for. Nothing in that file is user-facing until you run Pro on a real workflow and watch what a long agent actually costs.

Do the arithmetic on the profiles. A 100-step Pro run at 15–40 seconds per step is somewhere between 25 minutes and just over an hour of wall clock, and every one of those steps is at least one vision model call carrying a screenshot. Flash is cheaper per step but loops harder, and because its context is compressed rather than truncated it will happily run past the turn limit you assumed was a ceiling. Whichever profile you pick, the model is the line item that scales with your test suite, not the licence or the hardware.

This is where a routing layer stops being an abstraction. A vision model driving a long Android session is a workload with two awkward properties: it is long-lived, and it is intolerant of a provider hiccup in the middle of step 74, because the run's context is on that provider's endpoint. Pointing ARTEMIS at a single OpenAI-compatible base URL and letting our failover move the run to another provider of the same model is the difference between a flaky test and a lost afternoon. Qwen3.8-Flash is the interesting candidate to try first — a 6B-active multimodal MoE with a 1M-token context, listed on our catalogue at $0.15 per million input tokens and $0.47 per million output, with cache reads at $0.018 and cache writes at $0.230. That cache-read price is the relevant one here, because a Pro run re-reads a growing context on every step.

Be precise about what that means, though: Qwen3.8-Flash is not on ARTEMIS's tested backend list, which names Gemini, Claude, GPT-4o and Qwen-VL. It is a model that plausibly fits the harness, and the harness is explicitly built to accept one. Nobody has published a benchmark of that pairing, and you should treat anyone who claims one as having invented it.

The roadmap, and how much of it is load-bearing

Four items are on the published roadmap: an Android Studio integration with in-editor debugging, test recording and device control; iOS support; on-device lightweight vision-language models for low-latency, privacy-first work; and real-time duplex voice interaction.

The first is the one to believe, because it is the natural next artefact from a Pixel test-engineering team and it has no research risk attached — the agent already drives a device, it just needs a panel in the IDE. iOS is a much larger claim than it looks from the outside: the whole locating strategy leans on Android's accessibility service, and iOS has no equivalent surface with the same permissions model, so expect a rewrite of the perception layer rather than a port. The on-device VLM item is the one worth watching closely, because it is the only item on the list that would remove the per-step API cost that currently dominates a large test suite — and it implies a small, fast, vision-capable model that can hold a UI task together, which is a much narrower target than "a small model that is good at tool use."

What to do with it this week

Clone it, run code>./start.sh/code> against one emulator, and give Flash a task your existing Espresso suite covers. That will tell you within an hour whether the dynamic-first locator survives your app's Compose surfaces, which is the question that decides whether any of the rest matters. Then run the same task on Pro and compare the two traces — the gap between 3–5 and 15–40 seconds per step is where your budget lives, and you cannot reason about ARTEMIS's cost without it.

While you are reading the code, read the headers. The September 12 commit that added the Minitap attribution is the most recent one in the repository, which means the file headers you are reading are four days old and the project is being actively repaired in public. That is not a reason to avoid it. It is a reason to check which version of the story you are holding before you quote a success rate in a slide.

Pointing ARTEMIS at a single OpenAI-compatible base URL and letting our failover move the run to another provider of the same model is the difference between a flaky test and a lost afternoon.