Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representations

2025-05-25

AI Summary

Prompt injection attacks pose a severe threat to the security of large language models (LLMs). Existing defenses based on instruction hierarchies (IH) typically inject privileged signals only at the input layer, leading to signal attenuation as representations propagate through deeper layers and hindering dynamic, token-level permission discrimination. To address this, we propose a **multi-layer instruction hierarchy injection mechanism**, which— for the first time—introduces layer-specific, trainable embeddings into intermediate transformer representations. This approach integrates inter-layer privilege enhancement with dynamic modulation of intermediate states, enabling deep-fidelity preservation and adaptive regulation of IH signals. Evaluated across diverse LLMs and training configurations, our method reduces the success rate of gradient-based prompt injection attacks by 1.6–9.2×, while incurring negligible degradation (<0.5%) in original task performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Prompt injection attacks are a critical security vulnerability in large language models (LLMs), allowing attackers to hijack model behavior by injecting malicious instructions within the input context. Recent defense mechanisms have leveraged an Instruction Hierarchy (IH) Signal, often implemented through special delimiter tokens or additive embeddings to denote the privilege level of input tokens. However, these prior works typically inject the IH signal exclusively at the initial input layer, which we hypothesize limits its ability to effectively distinguish the privilege levels of tokens as it propagates through the different layers of the model. To overcome this limitation, we introduce a novel approach that injects the IH signal into the intermediate token representations within the network. Our method augments these representations with layer-specific trainable embeddings that encode the privilege information. Our evaluations across multiple models and training methods reveal that our proposal yields between $1.6 imes$ and $9.2 imes$ reduction in attack success rate on gradient-based prompt injection attacks compared to state-of-the-art methods, without significantly degrading the model's utility.

Problem

• Defending against prompt injection attacks in LLMs
• Enhancing Instruction Hierarchy signal propagation
• Reducing attack success without utility degradation

Innovation

• Augments intermediate token representations with IH
• Uses layer-specific trainable privilege embeddings
• Reduces attack success rate significantly