What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics

2026-08-17

AI Summary

This study addresses the limitation that context compression may preserve task completion rates while obscuring the costs of state loss. We propose a controlled runtime measurement protocol that decomposes tool invocations within fixed-step planning environments and incorporates oracle interventions to quantify the implicit interaction costs incurred when agents recover lost states. Empirical results demonstrate that although completion rates remain unchanged, compression induces a significant increase in retrieval calls, revealing efficiency degradation masked by traditional metrics and highlighting its environment-dependent nature. These findings establish that task completion alone is insufficient for evaluating the true impact of compression on agent efficiency, thereby offering novel perspectives and evaluation benchmarks for optimizing context management strategies in autonomous agents.

Abstract

Task completion is the standard metric for evaluating context compression, yet it is incomplete: compression can increase an agent's interaction cost by forcing it to reacquire dropped state while leaving completion statistically unchanged. We introduce a controlled runtime measurement protocol for reacquisition cost in a bounded-horizon tool-using agent. The agent acts in a deterministic planning environment under a fixed 24-turn horizon. We vary compression severity, compare a dropping operator with a fact-preserving operator, restore dropped state through controlled oracle interventions, and decompose tool calls into retrieval and execution. We evaluate three models across two task regimes. Retrieval calls increase in all six model-regime comparisons and account for almost all added interaction; five of six remain significant after Holm correction. At the prespecified 5x comparison point, completion changes are not significant in any cell. DeepSeek shows a significant completion drop only at 10x compression. GPT-5.5 is the clearest case: completion changes from 80% to 85% (p = 1.0) while retrieval increases from 21.0 to 63.9 calls (p = .002). Retention interventions further separate state quantity, state type, and content validity. Random selection is comparable to an offline hindsight oracle, while replacing retained D-state with semantically irrelevant content increases retrieval by 57% (p < .001) without a significant completion change. In a second environment, ALFWorld, sliding compression produces no retrieval surge, showing that the reacquisition signature is environment-dependent rather than intrinsic to shortening context. Overall, compression can impose hidden interaction costs when execution-relevant state becomes absent and must be reacquired, while completion alone may not expose those costs.

Problem

• Context Compression
• Interaction Cost
• Task-Completion Metrics
• State Reacquisition
• Agent Evaluation

Innovation

• Context Compression
• Interaction Cost
• Reacquisition Cost
• Runtime Measurement Protocol
• Tool-Using Agent