Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents

2026-03-30

AI Summary

This work addresses a critical gap in the evaluation of large language model (LLM) agents, which has predominantly focused on single-task success rates while neglecting reliability over extended, repeated execution. The study proposes the first reliability assessment framework tailored for long-horizon tasks, introducing four novel metrics—reliability decay curves, variance amplification factor, graceful degradation score, and collapse onset point—to establish reliability as a core dimension distinct from raw capability. Through large-scale experiments encompassing 10 models, 23,392 task replays, and 396 cross-domain tasks, combined with hierarchical time-bucket design and statistical analysis, the authors reveal significant discrepancies between model capability and reliability rankings, find that state-of-the-art models exhibit collapse rates as high as 19%, demonstrate that memory scaffolds commonly impair long-term performance, and uncover domain-specific patterns of reliability decay.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to this divergence. We introduce a reliability science framework for long-horizon LLM agents with four metrics: Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP). We evaluate 10 models across 23,392 episodes on a 396-task benchmark spanning four duration buckets and three domains. Key findings: (1) reliability decay is domain-stratified -- SE GDS drops from 0.90 to 0.44 while document processing is nearly flat (0.74 to 0.71); (2) VAF bifurcates by capability tier -- high VAF is a capability signature, not an instability signal; (3) capability and reliability rankings diverge substantially, with multi-rank inversions at long horizons; (4) frontier models have the highest meltdown rates (up to 19%) because they attempt ambitious multi-step strategies that sometimes spiral; and (5) memory scaffolds universally hurt long-horizon performance across all 10 models. These results motivate reliability as a first-class evaluation dimension alongside capability.

Problem

• reliability
• long-horizon tasks
• LLM agents
• capability-reliability divergence
• evaluation metrics

Innovation

• Reliability Science
• Long-Horizon LLM Agents
• Reliability Decay Curve
• Graceful Degradation Score
• Meltdown Onset Point