Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy

2024-10-09

arXiv.org

AI Summary

Large language models (LLMs) lack explicit instruction-level hierarchy, rendering them vulnerable to prompt injection and adversarial user requests—low-priority user inputs may overwrite critical system instructions. Method: We propose an instruction segmentation embedding mechanism that explicitly models hierarchical relationships and priority ordering among instruction types (e.g., system messages, user prompts) at the model architecture level. This is the first approach to directly encode instruction priority into input representations. Our method integrates BERT-style segment embeddings, instruction-type tokenization, and hierarchical attention. We further introduce two structured instruction benchmarks—Structured Query (SQ) and Instruction Hierarchy (IH)—and extend AlpacaEval for systematic evaluation. Results: Experiments show robust accuracy improvements of 15.75% on SQ and 18.68% on IH, alongside a 4.1% gain in instruction-following capability on AlpacaEval, demonstrating end-to-end, generalizable safety enhancement.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) are susceptible to security and safety threats, such as prompt injection, prompt extraction, and harmful requests. One major cause of these vulnerabilities is the lack of an instruction hierarchy. Modern LLM architectures treat all inputs equally, failing to distinguish between and prioritize various types of instructions, such as system messages, user prompts, and data. As a result, lower-priority user prompts may override more critical system instructions, including safety protocols. Existing approaches to achieving instruction hierarchy, such as delimiters and instruction-based training, do not address this issue at the architectural level. We introduce the Instructional Segment Embedding (ISE) technique, inspired by BERT, to modern large language models, which embeds instruction priority information directly into the model. This approach enables models to explicitly differentiate and prioritize various instruction types, significantly improving safety against malicious prompts that attempt to override priority rules. Our experiments on the Structured Query and Instruction Hierarchy benchmarks demonstrate an average robust accuracy increase of up to 15.75% and 18.68%, respectively. Furthermore, we observe an improvement in instruction-following capability of up to 4.1% evaluated on AlpacaEval. Overall, our approach offers a promising direction for enhancing the safety and effectiveness of LLM architectures.

Problem

• LLMs lack instruction hierarchy, causing security vulnerabilities.
• Existing methods fail to prioritize instruction types effectively.
• ISE embeds priority info, improving safety and instruction-following.

Innovation

• Instructional Segment Embedding (ISE) technique
• Embeds instruction priority directly into model
• Improves safety against malicious prompts