MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop

2026-06-21

AI Summary

This work addresses the limitations of existing macOS agent benchmarks, which lack support for modern frameworks and rely on binary pass/fail evaluations that fail to capture partial progress in long-horizon, multi-application tasks. To overcome these shortcomings, the authors introduce MacAgentBench, a comprehensive benchmark comprising 676 tasks spanning 25 macOS applications, with nearly 60% integrating both GUI and CLI interactions. MacAgentBench features a novel evaluation framework that supports framework augmentation and fine-grained capability annotation, employing rule-based deterministic assessment combined with multi-checkpoint scoring to move beyond simplistic binary outcomes. Experiments across 16 models and 3 frameworks show that Claude Opus 4.6 paired with OpenClaw achieves a 73.7% Pass@1 score, while fine-grained metrics reveal substantial variation in subgoal completion even among runs with identical Pass@1 results.

Abstract

Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs such as OpenClaw on Mac Mini for always-on automation. However, existing benchmarks, including those for macOS, evaluate agents without framework augmentation and rely on binary evaluation. As a result, they fail to capture both the framework capabilities leveraged by modern CUAs and the partial progress on long-horizon, multi-application tasks. We present MacAgentBench, a comprehensive macOS agent benchmark comprising 676 tasks across 25 applications, with nearly 60% involving both GUI and CLI interaction. The benchmark adopts deterministic rule-based evaluation and introduces fine-grained multi-checkpoint scoring with capability annotations for multi-application tasks. Experiments across three frameworks and 16 models show that the best configuration, Claude Opus 4.6 on OpenClaw, attains 73.7% Pass@1, while this advantage is primarily driven by the skill library rather than by framework design. Fine-grained metrics further reveal that models with similar Pass@1 can differ substantially in sub-goal completion. Our code and data are publicly available at https://github.com/JetAstra/MacAgentBench.

Problem

• AI agents
• desktop automation
• benchmarking
• macOS
• multi-application tasks

Innovation

• MacAgentBench
• desktop automation
• fine-grained evaluation
• multi-application tasks
• GUI-CLI interaction