Design Patterns for Securing LLM Agents against Prompt Injections

2025-06-10

AI Summary

Large language model (LLM) agents are vulnerable to prompt injection attacks, particularly during tool invocation and sensitive data handling, posing critical security risks. Method: This paper introduces the first provably secure design pattern system specifically for prompt injection defense. We formalize security requirements, conduct rigorous threat modeling, and abstract seven core defense patterns—covering input sanitization, context isolation, execution sandboxing, and other critical mitigation pathways—while formally quantifying the trade-offs between security guarantees and functional utility for each pattern. Contribution/Results: All patterns are validated on real-world, industrial-grade agent architectures. Empirical evaluation demonstrates that they effectively block high-severity prompt injection attacks and significantly enhance system robustness without compromising operational functionality.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Large-scale security measurements

Abstract

As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge. Among the most pressing threats are prompt injection attacks, which exploit the agent's resilience on natural language inputs -- an especially dangerous threat when agents are granted tool access or handle sensitive information. In this work, we propose a set of principled design patterns for building AI agents with provable resistance to prompt injection. We systematically analyze these patterns, discuss their trade-offs in terms of utility and security, and illustrate their real-world applicability through a series of case studies.

Problem

• Securing LLM agents against prompt injection attacks
• Designing AI agents with provable resistance to injections
• Balancing utility and security in agent design patterns

Innovation

• Design patterns for secure LLM agents
• Proven resistance to prompt injections
• Balancing utility and security trade-offs