Agentic Frameworks for Reasoning Tasks: An Empirical Study

2026-04-17

AI Summary

Existing agent frameworks lack systematic evaluation in terms of performance, efficiency, and practicality on complex reasoning tasks. This work presents the first unified empirical comparison of 22 prominent frameworks across three major reasoning benchmarks—BBH, GSM8K, and ARC—assessing accuracy, execution time, API cost, and failure-handling mechanisms. Drawing from an initial pool of 1,200 GitHub repositories, we introduce a taxonomy based on architectural design and demonstrate that orchestration quality significantly influences performance. Nineteen frameworks completed all evaluations, with twelve exhibiting consistent results (average accuracy: 74.6–75.9%; per-task latency: 4–6 seconds; cost: $0.0014–$0.0018). Notably, mathematical reasoning (GSM8K) remains challenging, achieving only 44.35% overall accuracy, and several frameworks suffered from inefficient resource usage or task failures due to suboptimal orchestration.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Planning, Routing, and Scheduling: Model-Based Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in agentic frameworks have enabled AI agents to perform complex reasoning and decision-making. However, evidence comparing their reasoning performance, efficiency, and practical suitability remains limited. To address this gap, we empirically evaluate 22 widely used agentic frameworks across three reasoning benchmarks: BBH, GSM8K, and ARC. The frameworks were selected from 1,200 GitHub repositories collected between January 2023 and July 2025 and organized into a taxonomy based on architectural design. We evaluated them under a unified setting, measuring reasoning accuracy, execution time, computational cost, and cross-benchmark consistency. Our results show that 19 of the 22 frameworks completed all three benchmarks. Among these, 12 showed stable performance, with mean accuracy of 74.6-75.9%, execution time of 4-6 seconds per task, and cost of 0.14-0.18 cents per task. Poorer results were mainly caused by orchestration problems rather than reasoning limits. For example, Camel failed to complete BBH after 11 days because of uncontrolled context growth, while Upsonic consumed USD 1,434 in one day because repeated extraction failures triggered costly retries. AutoGen and Mastra also exhausted API quotas through iterative interactions that increased prompt length without improving results. We also found a sharp drop in mathematical reasoning. Mean accuracy on GSM8K was 44.35%, compared with 89.80% on BBH and 89.56% on ARC. Overall, this study provides the first large-scale empirical comparison of agentic frameworks for reasoning-intensive software engineering tasks and shows that framework selection should prioritize orchestration quality, especially memory control, failure handling, and cost management.

Problem

• agentic frameworks
• reasoning tasks
• empirical evaluation
• computational cost
• orchestration quality

Innovation

• agentic frameworks
• empirical evaluation
• reasoning benchmarks
• orchestration quality
• computational cost