UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks

2026-07-09

AI Summary

Existing evaluation benchmarks for active agents predominantly rely on static sandbox environments and single-turn tasks, which hinder the disentanglement of complex capabilities and the identification of failure sources. This work proposes the first capability-driven evaluation framework for active agents, centered on five core competencies: skill utilization, exploration, long-context reasoning, multimodal understanding, and cross-platform coordination. The framework comprises 400 bilingual, real-world tasks evaluated through fine-grained step-wise checkpoints within live Docker containers, enabling closed-loop, multi-turn assessment. By decoupling the contributions of base models and agent architectures, it supports multi-agent collaboration—encompassing executors, supervisors, and users—and facilitates cross-framework model comparisons. This systematic approach reveals the joint impact of model and architecture on task performance, offering the community a reproducible and extensible evaluation platform.

Abstract

The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.

Problem

• proactive agents
• real-world tasks
• benchmark evaluation
• capability disentanglement
• multimodal large language models

Innovation

• capability-driven benchmark
• proactive agents
• real-world tasks
• closed-loop evaluation
• multimodal large language models