DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows

2026-08-26

AI Summary

针对现实世界复杂工作流中自主代理的评估问题，通过构建包含200个任务的DuMateBench基准，并在模拟真实环境复杂性的条件下测试不同框架和语言模型组合的表现。

Abstract

Autonomous agents are increasingly adopted to complete complex, multi-tool workflows in real-world settings. However, existing benchmarks typically separate tasks by application or capability and evaluate agents in environments that are cleaner and more stable than those encountered in practice. We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions collected from a large-scale production agent platform. Each task preserves the relevant pre-solution interaction history, persistent configurations, and workspace state, and is then validated through human verification. The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories, with most tasks requiring multiple capability coordination. We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy, and assess performance using a hybrid deterministic and LLM-as-Judge evaluation protocol. Experiments across five representative autonomous-agent frameworks paired with four state-of-the-art LLMs reveal substantial gaps in strict task completion. Complementary robustness, efficiency, and diagnostic analyses further show that performance under environmental perturbations is jointly shaped by the capabilities of the LLM and the surrounding agent framework. The code and data are publicly available at https://dumatebench.com/.

Problem

• Autonomous Agents
• Real-World Workflows
• Benchmarks
• Environmental Complexity

Innovation

• DuMateBench
• real-world workflows
• environmental complexity
• LLM-as-Judge