GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

2026-06-22

AI Summary

This work addresses the lack of fair comparison between graphical user interface (GUI) and command-line interface (CLI) agents due to confounding variables such as interaction modalities, task definitions, initial states, action spaces, and verification mechanisms. To isolate performance bottlenecks, the authors construct a matched benchmark comprising 440 desktop tasks spanning 18 applications and 12 workflow categories, enforcing identical task objectives, initial conditions, and final-state validation while restricting agents to their native action sets. Under this rigorously controlled setting, GUI agents are found to be limited by the reliability of long-horizon visual interactions, whereas CLI agents suffer from insufficient skill-interface coverage. Based on this insight, the paper introduces a verifier-guided skill augmentation method. Experiments show that the best GUI agent achieves a 59.1% full-task success rate, the baseline CLI agent 48.2%, and the augmented CLI agent improves to 69.3%, confirming skill coverage as the critical limiting factor.

Abstract

Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goals, states, and final-state verifiers while being restricted to modality-native actions. In this controlled setting, the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%; however, verifier-guided skill augmentation raises CLI success to 69.3%, showing that much of the CLI deficit comes from incomplete skill coverage rather than model capability alone. These results suggest that GUI and CLI expose different execution bottlenecks: GUI agents are limited by reliable grounded interaction over long-horizon workflows, whereas CLI agents are limited by the coverage and scalability of their skill interfaces.

Problem

• GUI
• CLI
• execution bottlenecks
• computer-use agents
• interaction modality

Innovation

• GUI vs. CLI
• computer-use agents
• matched benchmark
• skill augmentation
• execution bottlenecks