TelemetrySuffBench: Is Agent Telemetry Sufficient for Failure-Origin Diagnosis?

2026-08-07

AI Summary

This work addresses the limitations of existing agent telemetry data, which, while useful for fault detection, struggles with precise root cause localization under insufficient evidence and lacks a reliable abstention mechanism. The authors introduce TelemetrySuffBench, a novel benchmark that decouples fault detection, source localization, and safe abstention for the first time. It employs multi-component delayed-binding trajectories, seven-factor telemetry masking, and ambiguous source pairs to systematically evaluate model capabilities. A unified protocol, candidate-set constraints, invalid-output statistics, and a frozen blind test set are implemented, along with an evidence-gating mechanism. Experiments reveal that source localization achieves Top-1 accuracy of 33.8%–97.2% with full telemetry but drops to ≤0.5% when using only metadata or standard views. Evidence gating reduces unjustified answers by 12.5–48.6 percentage points, highlighting a significant performance gap between detection and localization and underscoring the critical role of decision content and provenance information in reliable attribution.

Abstract

Agent systems increasingly expose execution traces, yet telemetry that reveals a failure may still be inadequate for identifying where that failure originated. We introduce TelemetrySuffBench, a controlled benchmark that separates failure detection, fault-origin localization, and safe abstention under insufficient evidence. The benchmark constructs canonical multi-component traces with delayed-binding faults and renders them as paired coarse views, seven-factor telemetry masks, and exact-equal ambiguous origin pairs. We evaluate five frontier language models using unified protocols, explicit candidate sets, invalid-output accounting, subgroup analyses, and a frozen blind holdout. With full telemetry, origin-step Top-1 accuracy ranges from 33.8% to 97.2% across models. Metadata, OpenTelemetry-compatible, and OpenInference-compatible views retain 99.5% to 100% detection F1 while limiting origin-step accuracy to at most 0.5%, exposing a robust detection-localization gap. Factor ablations further show that removing decision content reduces origin-step accuracy to zero for every model, while provenance removal also causes large model-dependent losses. On rich ambiguous inputs that require abstention, evidence gating reduces unsupported unique-origin answers by 12.5 to 48.6 percentage points for three models, whereas two models still answer every case, revealing strong model dependence in safe abstention. Results on the frozen holdout reproduce the central pattern within the same generator family. These findings show that terminal status can support detection, whereas reliable causal attribution requires explicit decision-to-provenance links and abstention safeguards that remain effective across models. The dataset and benchmark implementation are available at https://anonymous.4open.science/r/TelemetrySuffBench-E635/README.md.

Problem

• agent telemetry
• failure-origin diagnosis
• fault localization
• telemetry sufficiency
• causal attribution

Innovation

• fault-origin localization
• agent telemetry
• safe abstention
• provenance-aware reasoning
• controlled benchmark