AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition

2026-02-11

AI Summary

Current tool-augmented large language model (LLM) agents perform well on idealized benchmarks but exhibit insufficient robustness in real-world noisy environments, and lack systematic evaluation methodologies. This work proposes AgentNoiseBench, the first framework to formally define and categorize environmental noise affecting agents into user-side and tool-side noise. It introduces a controllable and scalable noise injection mechanism and constructs an automated evaluation pipeline that preserves task solvability. Extensive experiments across diverse LLM architectures and scales reveal significant performance degradation under various noise conditions, highlighting the sensitivity of existing agents to realistic perturbations. AgentNoiseBench thus establishes the first systematic benchmark for robust agent research and offers clear directions for future improvements.

Technology Category

• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent advances in large language models have enabled LLM-based agents to achieve strong performance on a variety of benchmarks. However, their performance in real-world deployments often that observed on benchmark settings, especially in complex and imperfect environments. This discrepancy largely arises because prevailing training and evaluation paradigms are typically built on idealized assumptions, overlooking the inherent stochasticity and noise present in real-world interactions. To bridge this gap, we introduce AgentNoiseBench, a framework for systematically evaluating the robustness of agentic models under noisy environments. We first conduct an in-depth analysis of biases and uncertainties in real-world scenarios and categorize environmental noise into two primary types: user-noise and tool-noise. Building on this analysis, we develop an automated pipeline that injects controllable noise into existing agent-centric benchmarks while preserving task solvability. Leveraging this pipeline, we perform extensive evaluations across a wide range of models with diverse architectures and parameter scales. Our results reveal consistent performance variations under different noise conditions, highlighting the sensitivity of current agentic models to realistic environmental perturbations.

Problem

• LLM agents
• robustness
• environmental noise
• benchmarking
• real-world deployment

Innovation

• AgentNoiseBench
• robustness evaluation
• environmental noise
• tool-using LLM agents
• noise injection