The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective

2025-06-04

AI Summary

This work exposes a system-level cost crisis induced by dynamic reasoning in LLM agents: multi-turn tool invocation and test-time scaling cause explosive resource consumption, increase latency variance by 2–5×, and raise energy consumption per unit accuracy by over 300%, with sharply diminishing marginal returns. Methodologically, it presents the first end-to-end systems evaluation from an AI infrastructure perspective—integrating performance monitoring, power modeling, inference trace tracking, and datacenter-scale workload simulation—and establishes a three-dimensional trade-off framework balancing accuracy, cost, and latency. Through controlled multi-agent architecture experiments (few-shot prompting, reflection depth, parallel reasoning), it empirically validates efficiency degradation patterns. Key contributions include: (1) identifying dynamic reasoning as a critical sustainability bottleneck for production deployment; and (2) proposing lightweight, deployment-oriented inference design principles to mitigate systemic cost escalation.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Large-language-model (LLM)-based AI agents have recently showcased impressive versatility by employing dynamic reasoning, an adaptive, multi-step process that coordinates with external tools. This shift from static, single-turn inference to agentic, multi-turn workflows broadens task generalization and behavioral flexibility, but it also introduces serious concerns about system-level cost, efficiency, and sustainability. This paper presents the first comprehensive system-level analysis of AI agents, quantifying their resource usage, latency behavior, energy consumption, and datacenter-wide power consumption demands across diverse agent designs and test-time scaling strategies. We further characterize how AI agent design choices, such as few-shot prompting, reflection depth, and parallel reasoning, impact accuracy-cost tradeoffs. Our findings reveal that while agents improve accuracy with increased compute, they suffer from rapidly diminishing returns, widening latency variance, and unsustainable infrastructure costs. Through detailed evaluation of representative agents, we highlight the profound computational demands introduced by AI agent workflows, uncovering a looming sustainability crisis. These results call for a paradigm shift in agent design toward compute-efficient reasoning, balancing performance with deployability under real-world constraints.

Problem

• Analyzing system-level cost and efficiency of dynamic AI agents
• Quantifying resource usage and sustainability impacts of agent designs
• Balancing accuracy-cost tradeoffs in compute-intensive agent workflows

Innovation

• Dynamic reasoning with external tools coordination
• System-level analysis of AI agent resource usage
• Compute-efficient reasoning for sustainability