The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation

2026-07-13

AI Summary

This work addresses a critical gap in the evaluation of large language model (LLM) agents: the neglect of persistent data—such as logs, snapshots, and checkpoints—retained on disk after execution, which renders storage overhead invisible. To remedy this, we introduce AgentFootprint, the first cross-framework benchmark that explicitly incorporates persistent storage as a core evaluation dimension. Our approach employs a serialization-aware metric suite to quantify total retention volume, channel composition, redundancy, growth characteristics, compressibility, and reconstructability of dialogue history. Combining fixed-trajectory replay, content-addressable storage, and multi-framework experiments across seven agent frameworks and 108 SWE-bench instances, our study reveals striking disparities: configurations achieving 100% task accuracy exhibit up to 15.7× differences in storage overhead; full-history retention demonstrates superlinear growth; and content-addressable storage reduces storage by 4.8–32.7× while preserving full reconstructability.

Abstract

LLM agent benchmarks measure task completion, reliability, and inference cost, but not the persistent data an agent run leaves on disk, including logs, context snapshots, checkpoints, and debug traces. We introduce AgentFootprint, a cross-framework benchmark of post-run agent storage footprint. Its serialization-aware metric suite measures total retention, channel composition, duplication, growth, compressibility, and conversation-history reconstructability. It addresses a measurement trap: naive byte-level measurement understates duplication by an order of magnitude because database paging and JSON escaping obscure repeated content. A fixed-trace control separates agent-generated logical volume from persistence-layer amplification: replaying the same trajectory through seven persisting frameworks yields a 6.7x spread. Under identical models, tools, and tasks, configurations with 100% accuracy differ by 15.7x in retained bytes, although their defaults support different recovery and audit capabilities. Three full-history configurations grow superlinearly on a repeated-observation stress task. Exported trajectories from 108 instance-normalized SWE-bench Verified submissions span three orders of magnitude per instance, with no detectable correlation with resolve rate. A content-addressed store reduces retention by 4.8x-32.7x while preserving every reconstructability score. These results establish persistent storage as a resource metric to report jointly with accuracy and reconstructability.

Problem

• storage footprint
• LLM agent evaluation
• persistent data
• serialization
• benchmarking

Innovation

• storage footprint
• LLM agent evaluation
• serialization-aware metrics
• content-addressed storage
• persistent data measurement