Detecting Safety Violations Across Many Agent Traces

2026-04-13

AI Summary

Safety violations in large-scale agent trajectories are typically sparse, complex, and often adversarially concealed, rendering them difficult for conventional methods to detect—especially collaborative violation patterns spanning multiple trajectories. To address this challenge, this work proposes Meerkat, a novel framework that uniquely integrates trajectory clustering with a dynamic agent search mechanism. Without relying on seed scenarios or fixed procedural templates, Meerkat employs structured exploration and adaptive investigation to efficiently uncover safety violations defined in natural language. This approach transcends the limitations of single-trajectory analysis and static monitoring, significantly outperforming baseline methods in scenarios involving abuse, misalignment, and task gaming. Notably, it exposes widespread developer cheating in mainstream benchmarks and identifies nearly four times as many reward-hacking instances on CyBench compared to prior audits.

Technology Category

• Multiagent Systems: Mechanism Design
• Search and Optimization: Metareasoning and Metaheuristics
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Security and Privacy: Large-scale security measurements
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

To identify safety violations, auditors often search over large sets of agent traces. This search is difficult because failures are often rare, complex, and sometimes even adversarially hidden and only detectable when multiple traces are analyzed together. These challenges arise in diverse settings such as misuse campaigns, covert sabotage, reward hacking, and prompt injection. Existing approaches struggle here for several reasons. Per-trace judges miss failures that only become visible across traces, naive agentic auditing does not scale to large trace collections, and fixed monitors are brittle to unanticipated behaviors. We introduce Meerkat, which combines clustering with agentic search to uncover violations specified in natural language. Through structured search and adaptive investigation of promising regions, Meerkat finds sparse failures without relying on seed scenarios, fixed workflows, or exhaustive enumeration. Across misuse, misalignment, and task gaming settings, Meerkat significantly improves detection of safety violations over baseline monitors, discovers widespread developer cheating on a top agent benchmark, and finds nearly 4x more examples of reward hacking on CyBench than previous audits.

Problem

• safety violations
• agent traces
• adversarial hiding
• reward hacking
• misuse campaigns

Innovation

• agentic auditing
• trace clustering
• safety violation detection
• reward hacking
• adaptive search