🤖 AI Summary
This work proposes an "adversarial hallucination hijacking" attack under a realistic threat model where no direct interaction channel exists between the adversary and the target large language model (LLM). By modeling the LLM’s tendency to hallucinate popular resource names, the method predicts and registers identifiers—such as domain names or skill names—that the model is likely to generate, then pre-deploys malicious prompts at those locations. When users inadvertently invoke these hallucinated resources, remote code execution is triggered. This study is the first to demonstrate that the predictability and cross-model transferability of LLM hallucinations can enable large-scale, undirected attacks, establishing a novel prompt injection paradigm that requires no direct model interaction. Experiments show hallucination-triggered exploitation rates of 85% in repository cloning and 100% in skill installation scenarios, with successful remote code execution demonstrated across multiple production-grade LLM agent systems.
📝 Abstract
The growing adoption of agentic LLM applications has introduced a new threat previously named as promptware. While prior work has established that adversaries can exploit direct channels to LLM applications to apply promptware under weak threat models, many applications do not provide any direct channels that could be exploited for prompt injection beyond the Internet. This raises a question: can attackers exploit LLM applications at scale without any direct channels in practical threat models? In this work, we show that the inherent tendency of LLMs to hallucinate resource identifiers can be exploited to amplify untargeted promptware attacks that pull adversarial prompts at scale and could be exploited to establish a botnet. We introduce adversarial hallucination squatting, a technique in which attackers identify trending resources (e.g., popular repositories, popular skills, etc.), compute the LLM distribution of hallucinations on the trending resource names, and preemptively register them to host adversarial prompts. By leveraging the predictability and transferability of hallucinations across foundational LLMs and to application layers, adversaries can significantly amplify the reach of untargeted promptware under weak threat models and establish a botnet by exploiting LLM applications to install a bot on the device that pulled the compromised hallucinated resource from the Inter. We empirically demonstrate that hallucinated resource generation occurs at high rates, up to 85% in repository cloning scenarios and up to 100% in skill installation, and that these hallucinations transfer between foundational models and different prompts. We demonstrate the practicality of adversarial hallucination squatting against various production LLM applications with integrated terminals in their set of tools, achieving remote tool execution and remote code execution.