Abstract illustration of a script on a host machine sending a dotted request outward to a remote model endpoint and receiving a block of generated code back, drawn as a schematic rather than as a depiction of any real system, organisation or person. The OrcaRouter logo is composited in the bottom-right corner.
AI Safety Incidents

PROMPTFLUX and PROMPTSTEAL: the first malware that asks an LLM what to do next

Author

jinhao.song

Date Published

Latest models · 20View all models →
Benchmarks: Artificial Analysis · updated daily
Back to all posts

On 5 November 2025, Google Threat Intelligence Group published "GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools", a report describing what it calls the first malware families to query a large language model while they are executing. One of them, PROMPTSTEAL, was used by the Russian state-backed group APT28 (also tracked as FROZENLAKE) against Ukrainian government targets, sending prompts to Qwen2.5-Coder-32B-Instruct through the Hugging Face API and running whatever Windows commands came back; Ukraine's CERT-UA had already documented the same family in July 2025 under the name LAMEHUG. The other, PROMPTFLUX, is an experimental VBScript dropper that asks the Gemini API to rewrite its own code for antivirus evasion, and GTIG states outright that it "does not demonstrate an ability to compromise a victim network or device". The harm in this record is the Ukrainian campaign; the novelty is that a model, rather than the malware author, supplied the commands. The full record, with its severity and confidence grades, sits in the Orca AI Incident Archive, alongside the neighbouring entries on malware that talks to a model mid-execution.

That distinction is the reason this is a landmark entry rather than another phishing write-up. Malware has used AI at authoring time for several years — a developer asking a chatbot for a script, then hard-coding the result. What GTIG documents is different in kind: the code that runs on the victim's machine is not fixed when the sample is compiled. It is fetched at execution time from a third-party commercial API, in the attacker's own prompt, and executed without a human reading it. GTIG labels the finding "just-in-time" AI and calls it "a significant step toward more autonomous and adaptive malware". It also calls the technique nascent, and the report's own status column is the most useful thing in it: of the five families it names, three — FRUITSHELL, PROMPTSTEAL and QUIETVAULT — are marked "Observed in operations", and two, PROMPTFLUX and PROMPTLOCK, are marked "Experimental".

What GTIG says it found

The report is an update to Google's January 2025 analysis of generative-AI misuse, and its evidence base is Google's own telemetry: prompts and activity it observed against Gemini, plus malware samples its researchers analysed. Table 1 of the report lists five code families with what GTIG calls "novel AI capabilities", each with a status of either "Observed in operations" or "Experimental". Only PROMPTSTEAL sits in the first category with a live, attributable campaign behind it. PROMPTFLUX, PROMPTLOCK and the others are catalogued as experimental or as tooling rather than as operations.

The mechanism in both headline families is the same shape. Instead of shipping a list of commands, the sample ships a prompt — base64-encoded text in PROMPTSTEAL's case, a machine-parsable instruction in PROMPTFLUX's — posts it to a public model API with a key embedded in the binary, and executes the response. Neither family needed a vulnerability in the model provider's service. Both used the API as designed, which is why GTIG frames the problem as abuse of a legitimate capability rather than a flaw that a patch closes.

A self-built comparison card headed “Two families, one archive record”, with two columns. Left column, PROMPTSTEAL, also reported as LAMEHUG: Python data miner packaged with PyInstaller; sends prompts to Qwen2.5-Coder-32B-Instruct through the Hugging Face API; asks for one-line Windows commands to collect system information and documents in named folders; executes the output locally and exfiltrates by SFTP or HTTP POST; poses as an image-generation program; used by APT28 against Ukraine; badge reads observed in operations. Right column, PROMPTFLUX: VBScript dropper with a Thinking Robot module; posts to the Gemini API with a hard-coded key and the gemini-1.5-flash-latest model tag; asks for VBScript obfuscation and antivirus-evasion code only; saves the regenerated script to the Windows Startup folder and copies itself to removable drives and mapped network shares; the AttemptToUpdateSelf routine is commented out; actor unattributed; badge reads experimental. The OrcaRouter logo is composited in the bottom-right corner.

PROMPTSTEAL: a real campaign with an unmeasured yield

CERT-UA got there first. On 17 July 2025, Ukraine's Computer Emergency Response Team published an advisory on a family it named LAMEHUG, after receiving reports on 10 July 2025 of suspicious emails sent from compromised accounts and impersonating ministry officials. The messages targeted "executive government authorities" and carried a ZIP archive containing three variants of the payload, named "Додаток.pif", "AI_generator_uncensored_Canvas_PRO_v0.9.exe" and "image.py". CERT-UA attributed the activity to APT28 with medium confidence — a lower grade than GTIG's flat assertion that the actor is Russian government-backed — and described the LLM's role in one line: it is "used to generate commands based on their textual representation (description)".

GTIG's account adds the operational detail. PROMPTSTEAL masquerades as an "image generation" program that walks the user through prompts while querying Hugging Face in the background — a disguise that fits the filenames CERT-UA recovered. The prompts GTIG recovered ask for system information and for documents in specific folders; the returned commands are, in its words, "blindly executed locally" before the output is exfiltrated. GTIG says the family "likely uses stolen API tokens" to reach Hugging Face, and that analysis shows continued development, with new samples adding obfuscation and changing the command-and-control method. A follow-up by Cato Networks, folded into The Hacker News's coverage on 25 July 2025, counted roughly 270 Hugging Face tokens used for authentication and noted that the malware carries pre-defined, base64-encoded descriptions of the objectives it wants commands for.

A self-built timeline card titled “How the record was built” listing six dated entries: early June 2025, GTIG identifies PROMPTFLUX, an experimental VBScript dropper calling the Gemini API; June 2025, GTIG identifies APT28 using PROMPTSTEAL against Ukraine; 10 July 2025, CERT-UA receives reports of phishing emails impersonating ministry officials; 17 July 2025, CERT-UA publishes its LAMEHUG advisory, linking the activity to APT28 with medium confidence; 25 July 2025, Cato Networks adds token and variant analysis as coverage is updated the same day; 5 November 2025, GTIG publishes the report naming both families and grading PROMPTFLUX as experimental. A footer notes that 5 November 2025 is the report's publication date, not the date of any intrusion. The OrcaRouter logo is composited in the bottom-right corner.

PROMPTFLUX: the one that was not used

PROMPTFLUX is where the reporting and the headlines diverge most. GTIG identified it in early June 2025: a VBScript dropper whose central component, a module the authors called "Thinking Robot", sends a POST request to the Gemini API endpoint with a hard-coded key, specifying the "gemini-1.5-flash-latest" model. The "-latest" tag matters — it means the malware follows whatever stable release Google currently serves, so the tool survives model deprecation. GTIG describes the prompt as "highly specific and machine-parsable, requesting VBScript code for antivirus evasion and instructing the LLM to output only the code itself". One variant goes further, asking the model to act as an "expert VBScript obfuscator" and rewrite the entire source hourly, re-embedding the decoy payload, the API key and the regeneration logic so the cycle can repeat.

Then the caveats, which are in the primary document and get lost in aggregation. The self-modification routine, AttemptToUpdateSelf, is commented out. A mechanism exists to limit how often the malware calls the API. AI responses are logged to a file, %TEMP%\thinking_robot_log.txt. GTIG reads those traces as intent rather than capability, concluding the family is "in a development or testing phase" and that "the current state of this malware does not demonstrate an ability to compromise a victim network or device". The author is unattributed; GTIG infers a financially motivated actor from lure filenames such as "crypted_ScreenRec_webinstall" and a broad, geography-agnostic targeting style. Google says it disabled the assets associated with the activity, and that DeepMind used the findings to strengthen both its classifiers and the model.

"I am working on a CTF problem"

The record's second theme is cheaper than any malware: talking the model into helping. GTIG describes a China-nexus actor that asked Gemini to identify vulnerabilities on a compromised system and received a safety refusal, then reframed the request as a capture-the-flag exercise — "I am working on a CTF problem", "I am currently in a CTF, and I saw someone from another team say …" — and got the exploitation guidance it wanted. GTIG says the actor learned from the exchange and reused the pretext across phishing, exploitation and web shell development. A separate Iranian actor, TEMP.Zagros (also tracked as MUDDYCOAST), used student pretexts: a final-year university project, "writing a paper", an "international article" on cybersecurity.

Google's response, as stated in the report, was to disable the accounts and feed the interactions back into training, and it is candid that the benign and malicious versions of these prompts are hard to tell apart: the same question from an actual CTF player is legitimate. One detail from the TEMP.Zagros case is worth carrying, because it is the clearest evidence in the document that runtime AI use cuts both ways. While debugging a script with Gemini, the actor pasted in hard-coded material including its command-and-control domain and the script's encryption key, which GTIG says fed its broader disruption of the campaign.

Established, contested, not established

The archive grades this record confidence A, not disputed, with real harm recorded, and the underlying sequence is indeed documented twice over by parties with independent access: CERT-UA from Ukrainian incident data, GTIG from Google's own product telemetry. Independent outlets — The Hacker News, SecurityWeek, Security Affairs, Wiz — restate the mechanics without contradicting them, and Wiz's own read of LAMEHUG matches CERT-UA's description of base64-encoded prompts and command execution. What is genuinely contested is significance, not fact, and it is worth separating the two.

Marcus Hutchins argued publicly, in comments reported by The Hacker News on 5 November 2025, that the PROMPTFLUX story is overblown. The prompt, he wrote, works "under the assumption that Gemini just instinctively knows how to evade antiviruses (it doesn't)", and there is "no entropy to ensure the 'self-modifying' code differs from previous versions, or any guardrails to ensure it actually works" — a technical objection that lands squarely on the fact that the update routine was disabled. Wiz reached a similar conclusion about runtime LLM use in general: attackers could have generated and tested the code beforehand, the model's output was inconsistent, and the approach left provider-side audit logs. Cato Networks, for its part, assessed the June activity as testing new capability in Ukraine rather than a full-fledged operation. None of that disputes that PROMPTSTEAL ran against real targets; all of it bears on how much the LLM added.

A self-built three-column card headed “Established, attributed, not established”. Column one, established: GTIG published the report on 5 November 2025; CERT-UA reported the same family as LAMEHUG on 17 July 2025 from Ukrainian incident data; PROMPTSTEAL queried Qwen2.5-Coder-32B-Instruct through the Hugging Face API and executed the returned commands; PROMPTFLUX calls the Gemini API with a hard-coded key and a gemini-1.5-flash-latest tag; PROMPTFLUX's self-update routine is commented out and Google states it cannot compromise a device. Column two, attributed or assessed: CERT-UA links the activity to APT28 with medium confidence; GTIG states the actor is Russian government-backed; the use of stolen Hugging Face tokens is GTIG's assessment; the financially motivated reading of PROMPTFLUX is inferred from lure filenames; the evasion aim is GTIG's “likely”. Column three, not established: how much data was exfiltrated and from how many victims; whether any LLM-generated command succeeded in an intrusion; who wrote or operated PROMPTFLUX; any independent telemetry beyond the two reporting organisations; any action by Hugging Face or by law enforcement. A footer notes that the archive grades the record confidence A and not disputed, which concerns the documentation rather than the debate over significance. The OrcaRouter logo is composited in the bottom-right corner.

What changed, and what the record does not show

The documented response is Google's. GTIG says it disabled the assets and accounts behind the Gemini misuse cases, and that DeepMind used the observations to strengthen both classifiers and the model so that it refuses these requests in future. It also points to its Secure AI Framework and to Big Sleep, its vulnerability-hunting agent, as the defensive side of the same coin. What the report does not record is any change at Hugging Face, any takedown of the PROMPTSTEAL infrastructure by a party other than the reporting researchers, or any law-enforcement action — and it does not put a number on the campaign: no victim count, no volume of stolen documents, no statement that the LLM-generated commands worked.

That gap is the honest shape of the record. A Russian state actor ran a data-theft campaign against Ukrainian government bodies in which the collection commands were written at runtime by a public coding model, and the malware was recovered and described in detail; how much it took is not in either advisory. A second family demonstrates that self-rewriting malware calling a commercial model API is buildable today, and its own authors had not yet switched it on. Both facts belong in the same entry because they arrived in the same report, and the report's own status column is what keeps them apart. Read that way, 5 November 2025 marks the date the technique stopped being hypothetical in one case and stayed hypothetical in the other.