AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

2026-07-15

AI Summary

Current evaluation frameworks for large language model agents suffer from fragmentation and tight coupling, lacking a unified and reproducible infrastructure. To address this, this work proposes a lightweight, open-source, and extensible evaluation framework that decouples the system into three modular components—Benchmark, Harness, and Environment—enabling highly flexible and reusable evaluation pipelines. The framework incorporates an asynchronous fault-tolerant runtime, trajectory logging, and analysis tools, and natively supports over 20 benchmarks spanning five core capability dimensions. This design facilitates transparent diagnosis of complex failure modes such as reward hacking. By significantly reducing engineering redundancy in evaluation workflows, this work advances the standardization and transparency of agent assessment.

Abstract

As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering. To address this, we introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. AgentCompass organizes the evaluation process around three independent components, namely Benchmark, Harness, and Environment, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic. Furthermore, it features a fault-tolerant asynchronous runtime and comprehensive trajectory analysis tools to transparently diagnose nuanced failure modes like reward-hacking. Natively supporting over 20 benchmarks across five capability dimensions, AgentCompass provides the community with a scalable and reproducible infrastructure for advancing agent research.

Problem

• evaluation infrastructure
• autonomous agents
• reproducibility
• fragmentation
• LLM-based agents

Innovation

• Agent Evaluation
• Modular Infrastructure
• Asynchronous Runtime
• Trajectory Analysis
• Reproducibility