TopicAttack: An Indirect Prompt Injection Attack via Topic Transition

2025-07-18

AI Summary

Large language models (LLMs) struggle to distinguish between instructions and data, rendering them vulnerable to indirect prompt injection attacks. To address this, we propose TopicAttack—a novel adversarial method based on progressive topic transition. Unlike conventional abrupt instruction injection, TopicAttack stealthily embeds malicious instructions into external data sources by fabricating natural conversational transitions, leveraging controllable text generation and attention-guided mechanisms to induce task deviation covertly. Its core innovation lies in constructing coherent, semantically plausible topic migration paths, significantly enhancing both attack deception and robustness. Experimental evaluations across multiple state-of-the-art LLMs—including those equipped with prominent defense mechanisms—demonstrate a success rate exceeding 90%. Attention visualization and ablation studies further confirm the efficacy of topic-guided instruction injection in evading detection and compromising model behavior.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) have shown remarkable performance across a range of NLP tasks. However, their strong instruction-following capabilities and inability to distinguish instructions from data content make them vulnerable to indirect prompt injection attacks. In such attacks, instructions with malicious purposes are injected into external data sources, such as web documents. When LLMs retrieve this injected data through tools, such as a search engine and execute the injected instructions, they provide misled responses. Recent attack methods have demonstrated potential, but their abrupt instruction injection often undermines their effectiveness. Motivated by the limitations of existing attack methods, we propose TopicAttack, which prompts the LLM to generate a fabricated conversational transition prompt that gradually shifts the topic toward the injected instruction, making the injection smoother and enhancing the plausibility and success of the attack. Through comprehensive experiments, TopicAttack achieves state-of-the-art performance, with an attack success rate (ASR) over 90% in most cases, even when various defense methods are applied. We further analyze its effectiveness by examining attention scores. We find that a higher injected-to-original attention ratio leads to a greater success probability, and our method achieves a much higher ratio than the baseline methods.

Problem

• Indirect prompt injection attacks exploit LLMs' instruction-following vulnerabilities
• Existing attacks fail due to abrupt topic transition in injections
• TopicAttack smooths malicious instruction injection via conversational topic shifts

Innovation

• Indirect prompt injection via topic transition
• Gradual topic shift for smoother attack
• High attack success rate over 90%