TrajAD: Trajectory Anomaly Detection for Trustworthy LLM Agents

2026-02-06

AI Summary

This work addresses a critical limitation in current safety mechanisms for large language model (LLM) agents, which predominantly rely on static input/output filtering and struggle to detect and precisely localize anomalies within intermediate execution trajectories, thereby compromising procedural reliability. To tackle this challenge, the study formally introduces the task of trajectory anomaly detection and proposes TrajBench, a fine-grained trajectory dataset synthesized via a perturbation-completion strategy. Building upon this, the authors develop TrajAD, a specialized verifier trained through fine-tuning to accurately identify anomalous steps in execution traces. Experimental results demonstrate that TrajAD substantially outperforms general-purpose LLM baselines, underscoring the pivotal role of dedicated process-level supervision in constructing trustworthy LLM agents and providing a foundational technical advance toward reliable autonomous systems.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We address the problem of runtime trajectory anomaly detection, a critical capability for enabling trustworthy LLM agents. Current safety measures predominantly focus on static input/output filtering. However, we argue that ensuring LLM agents reliability requires auditing the intermediate execution process. In this work, we formulate the task of Trajectory Anomaly Detection. The goal is not merely detection, but precise error localization. This capability is essential for enabling efficient rollback-and-retry. To achieve this, we construct TrajBench, a dataset synthesized via a perturb-and-complete strategy to cover diverse procedural anomalies. Using this benchmark, we investigate the capability of models in process supervision. We observe that general-purpose LLMs, even with zero-shot prompting, struggle to identify and localize these anomalies. This reveals that generalized capabilities do not automatically translate to process reliability. To address this, we propose TrajAD, a specialized verifier trained with fine-grained process supervision. Our approach outperforms baselines, demonstrating that specialized supervision is essential for building trustworthy agents.

Problem

• trajectory anomaly detection
• LLM agents
• process supervision
• error localization
• trustworthy AI

Innovation

• Trajectory Anomaly Detection
• Process Supervision
• LLM Agent Trustworthiness
• Error Localization
• TrajBench