Exploit Tool Invocation Prompt for Tool Behavior Hijacking in LLM-Based Agentic System

2025-09-06

AI Summary

This paper identifies, for the first time, critical security vulnerabilities in Tool Invocation Prompts (TIPs) within large language model (LLM) agent systems—specifically, their susceptibility to remote code execution (RCE) and denial-of-service (DoS) attacks. Method: We introduce “Tool Behavior Hijacking,” a novel attack paradigm, and formalize a systematic TIP Exploitation Workflow (TEW) that achieves unauthorized external tool invocation via semantic manipulation of prompts. Contribution/Results: We empirically validate the attack’s universality and severity across mainstream LLM agents—including Cursor and Claude Code—demonstrating practical exploitability. Furthermore, we propose a lightweight defense mechanism that blocks hijacking paths without modifying the underlying LLM, significantly enhancing TIP security. This work establishes the first threat model and defense framework specifically targeting the tool invocation layer in LLM agent systems, advancing the state of LLM agent security.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

LLM-based agentic systems leverage large language models to handle user queries, make decisions, and execute external tools for complex tasks across domains like chatbots, customer service, and software engineering. A critical component of these systems is the Tool Invocation Prompt (TIP), which defines tool interaction protocols and guides LLMs to ensure the security and correctness of tool usage. Despite its importance, TIP security has been largely overlooked. This work investigates TIP-related security risks, revealing that major LLM-based systems like Cursor, Claude Code, and others are vulnerable to attacks such as remote code execution (RCE) and denial of service (DoS). Through a systematic TIP exploitation workflow (TEW), we demonstrate external tool behavior hijacking via manipulated tool invocations. We also propose defense mechanisms to enhance TIP security in LLM-based agentic systems.

Problem

• Investigates security risks in Tool Invocation Prompts
• Reveals vulnerabilities enabling remote code execution attacks
• Addresses tool behavior hijacking in LLM-based systems

Innovation

• Tool Invocation Prompt exploitation workflow
• Demonstrates external tool behavior hijacking
• Proposes defense mechanisms for TIP security