Progent: Programmable Privilege Control for LLM Agents

2025-04-16

AI Summary

LLM agents are vulnerable to adversarial instructions during external interactions, potentially triggering unauthorized actions and posing severe security risks. To address this, we propose the first programmable least-privilege control framework tailored for LLM agents. Our approach leverages a domain-specific language (DSL) to enforce fine-grained, runtime constraints on tool invocation and provide deterministic fallback mechanisms. It supports user-defined policies, modular integration, and automatic policy generation and dynamic updating by the LLM itself. Balancing security and practicality, our method significantly improves robustness against adversarial attacks across three major benchmarks—AgentDojo, ASB, and AgentPoison—while maintaining high task success rates. Empirical evaluation demonstrates its effectiveness in defending against adaptive attacks, validating both its security guarantees and operational viability.

Technology Category

• Multiagent Systems: Adversarial Agents
• Natural Language Processing: Safety and Robustness
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

LLM agents are an emerging form of AI systems where large language models (LLMs) serve as the central component, utilizing a diverse set of tools to complete user-assigned tasks. Despite their great potential, LLM agents pose significant security risks. When interacting with the external world, they may encounter malicious commands from attackers, leading to the execution of dangerous actions. A promising way to address this is by enforcing the principle of least privilege: allowing only essential actions for task completion while blocking unnecessary ones. However, achieving this is challenging, as it requires covering diverse agent scenarios while preserving both security and utility. We introduce Progent, the first privilege control mechanism for LLM agents. At its core is a domain-specific language for flexibly expressing privilege control policies applied during agent execution. These policies provide fine-grained constraints over tool calls, deciding when tool calls are permissible and specifying fallbacks if they are not. This enables agent developers and users to craft suitable policies for their specific use cases and enforce them deterministically to guarantee security. Thanks to its modular design, integrating Progent does not alter agent internals and requires only minimal changes to agent implementation, enhancing its practicality and potential for widespread adoption. To automate policy writing, we leverage LLMs to generate policies based on user queries, which are then updated dynamically for improved security and utility. Our extensive evaluation shows that it enables strong security while preserving high utility across three distinct scenarios or benchmarks: AgentDojo, ASB, and AgentPoison. Furthermore, we perform an in-depth analysis, showcasing the effectiveness of its core components and the resilience of its automated policy generation against adaptive attacks.

Problem

• Addressing security risks in LLM agents from malicious commands
• Enforcing least privilege for safe tool calls in diverse scenarios
• Automating policy generation to balance security and utility

Innovation

• Domain-specific language for privilege control
• Modular design for easy integration
• LLM-generated dynamic policy updates