Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback

2026-05-17

AI Summary

This study addresses the security risk of "cognitive poisoning" in large language model agents, where malicious external tools exploit stealthy trigger mechanisms during tool invocation. It presents the first systematic investigation of attacks and defenses under untrusted tool feedback. The authors introduce TRUST-Bench, a new benchmark, along with the GuardedJoint evaluation metric, and propose VISTA-Guard, a general-purpose defense framework. VISTA-Guard abstracts multi-step tool interaction trajectories through structured environmental variables to model dynamic trust and compute trajectory-aware risk scores for final actions. Moving beyond conventional approaches that rely solely on prompts or tool descriptions, the method achieves GuardedJoint scores of 84.2 in-domain and 56.9 on balanced out-of-domain evaluations, substantially outperforming existing baselines.

Abstract

Tool-using LLM agents increasingly rely on external tools to make consequential decisions, yet most existing agent-security benchmarks and defenses implicitly assume that tool feedback is trustworthy once a tool has been selected. We study a different failure mode, cognitive poisoning, in which a malicious tool behaves plausibly during exploration, accumulates trust through benign-looking feedback, and becomes harmful only when hidden state conditions align with the final executable action. To study this setting, we construct TRUST-Bench, a task-conditioned benchmark of 1,970 hidden-trigger tool-compromise episodes with matched safe controls, introduce an asymmetric penalty metric, GuardedJoint, to better reflect real deployment risk, and present VISTA-Guard, a backbone-agnostic framework for final-action risk scoring. The core idea is to abstract multi-step tool interaction into structured environment variables that encode trust-formation dynamics and then score the risk of the final executable action from this trajectory-conditioned representation. Experiments show that prompt-centric heuristics, scalarized features, and zero-shot judges fail in this regime, whereas trajectory-aware final-action scoring yields strong in-domain discrimination and remains effective under balanced out-of-distribution transfer. Under GuardedJoint, VISTA-Guard reaches $84.2$ in-domain and $56.9$ on balanced out-of-distribution evaluation, while methods that optimize only one side of the safety--utility tradeoff collapse to zero. These findings support a broader view of agent security in black-box tool ecosystems: the decisive defense target is not local prompt text or tool descriptors alone, but the way trust is formed across the interaction trajectory and committed through the final action.

Problem

• tool feedback trustworthiness
• cognitive poisoning
• LLM agent security
• hidden-trigger attacks
• trust formation

Innovation

• cognitive poisoning
• tool-use security
• trajectory-aware risk scoring
• LLM agents
• trust dynamics