prompt engineering

The absence of standardized, adaptive evaluation frameworks for prompt injection attacks hinders rigorous assessment of LLM robustness. Method: This paper introduces the first optimization-based toolkit for evaluating prompt injection attacks and defenses in LLMs. It innovatively integrates white-box gradient optimization with query-efficient black-box optimization (e.g., Bayesian optimization) to construct a schedulable, modular, adaptive red-teaming framework capable of generating worst-case prompt samples and conducting stringent robustness evaluations. Contribution/Results: It is the first to enable adaptive, hybrid white-box/black-box attack generation; establishes an open-source, reproducible standardized evaluation pipeline (OET); and empirically demonstrates significant vulnerabilities in state-of-the-art defense mechanisms—several hardened models remain jailbreakable under OET, validating the framework’s rigor and practical utility.

This work proposes Image Prompt Injection (IPI), a novel attack method that exploits the vulnerability of multimodal large language models (MLLMs) to adversarial instructions embedded in images, enabling malicious manipulation of model behavior. IPI establishes the first end-to-end, image-level prompt injection framework under black-box settings, leveraging segmentation-based region selection, adaptive font scaling, and background-aware rendering to ensure injected prompts remain imperceptible to humans while remaining interpretable by the model. Experimental evaluation on the COCO dataset with GPT-4 Turbo demonstrates that IPI achieves a 64% attack success rate under strict visual stealth constraints, highlighting its effectiveness and potential threat in real-world scenarios.

This work addresses the vulnerability of large language models (LLMs) in integrated applications to prompt injection attacks, where malicious instructions—often injected through multiple sources—exhibit ambiguous semantic boundaries with legitimate context, rendering them difficult to detect. To counter this threat, the authors propose IntrusCoT, a novel approach that uniquely integrates instruction-level chain-of-thought learning with diverse adversarial data synthesis. Through fine-tuning, IntrusCoT enhances the model’s ability to recognize and refuse covertly embedded malicious instructions. Experimental results demonstrate that the method significantly outperforms existing baselines across four mainstream LLMs, achieving strong defensive performance against three critical risks: behavioral deviation, privacy leakage, and harmful outputs—while preserving the models’ original task capabilities.

This work identifies and exploits quasi-loss signals returned by fine-tuning APIs of closed-source large language models (e.g., Google Gemini) to devise a novel black-box prompt injection attack. Unlike conventional black-box methods, it requires no access to model weights or gradients; instead, it systematically reverse-engineers side-channel information from fine-tuning API responses—interpreting them as discrete optimization feedback—and integrates this signal into a greedy search framework for adversarial prompt generation. Its key contribution lies in the first principled use of fine-grained loss-like feedback for efficient, parameter-free prompt optimization. Experiments on the PurpleLlama benchmark demonstrate attack success rates of 65%–82% across Gemini models, revealing fine-tuning APIs as a previously overlooked attack surface for prompt injection. This finding delivers both a critical security warning and a new technical paradigm for API security auditing and red-teaming evaluations.

Prompt injection attacks pose a severe threat to the security of large language models (LLMs). Existing defenses based on instruction hierarchies (IH) typically inject privileged signals only at the input layer, leading to signal attenuation as representations propagate through deeper layers and hindering dynamic, token-level permission discrimination. To address this, we propose a **multi-layer instruction hierarchy injection mechanism**, which— for the first time—introduces layer-specific, trainable embeddings into intermediate transformer representations. This approach integrates inter-layer privilege enhancement with dynamic modulation of intermediate states, enabling deep-fidelity preservation and adaptive regulation of IH signals. Evaluated across diverse LLMs and training configurations, our method reduces the success rate of gradient-based prompt injection attacks by 1.6–9.2×, while incurring negligible degradation (<0.5%) in original task performance.

This work addresses the limited generalization of current large language models in defending against prompt injection attacks, particularly their vulnerability to subtle attacks that produce outputs nearly correct yet fundamentally flawed. To enhance robustness, the authors propose an alignment training method based on automatically generated “near-target” adversarial examples. This approach incorporates a margin-aware weighting mechanism that dynamically adjusts sample weights during training and leverages prompt engineering to enable single-step adversarial example generation. By sharpening the semantic boundary between instruction and data regions, the method significantly improves the model’s resilience and generalization against stealthy prompt injection attacks, outperforming existing defense strategies in real-world scenarios.

Existing prompt injection detection methods rely on large neural networks, which struggle to meet the stringent requirements of first-layer defenses—namely low latency, determinism, prompt-immunity, and auditability. To address this, this work proposes the Mirror design pattern, which constructs a strictly paired positive–negative sample mirror topology to steer classifiers toward learning the underlying attack mechanisms rather than exploiting data shortcuts, thereby prioritizing data organization over model scale. Using 5,000 open-source samples, the authors build a 32-cell mirror dataset to train a sparse character n-gram linear SVM, which is then compiled into a static Rust module. Evaluated on a held-out test set of 524 examples, the approach achieves a 95.97% recall and 92.07% F1 score, with inference latency under 1 millisecond and no dependency on external models.

Although large language models (LLMs) exhibit remarkable capabilities, they remain vulnerable to adversarial prompt attacks that can circumvent alignment-based defenses. This work presents the first formal game-theoretic model of the strategic interaction between attackers and defenders in this context, revealing an inherent advantage for the attacker. By integrating game theory, adversarial prompt modeling, and equilibrium analysis, the study derives theoretically provable optimal strategies for both attack and defense. Experimental results across multiple mainstream LLMs and benchmark datasets demonstrate that the proposed theoretically optimal attack strategy substantially outperforms existing methods, offering a rigorous theoretical foundation and practical guidance for the secure deployment of LLMs.