Problem
Research questions and friction points this paper is trying to address.
terminal-use agents
benchmark
general-purpose agents
computer-use tasks
evaluation
TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
🤖 AI Summary
Existing benchmarks inadequately evaluate the capabilities of general-purpose terminal agents on non-programming, everyday digital tasks due to a lack of comprehensive assessment frameworks that encompass realistic and diverse terminal interactions. This work proposes the first evaluation framework tailored to general terminal usage scenarios, covering both routine and professional tasks such as document editing, email management, web search, and research-oriented engineering workflows. The framework leverages authentic terminal environments, human-designed tasks, deterministic execution scripts, and outcome-based scoring protocols to enable reproducible evaluation. Experimental results demonstrate that even the current strongest model—Claude Code combined with Claude Opus 4.8—achieves only a 65.8% overall success rate, revealing substantial performance gaps in handling general terminal tasks.
📝 Abstract
As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities-including document editing, email management, and live-web information seeking-as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Terminal-Use Agents
General-Purpose Benchmark
Execution-Based Evaluation
Real-World Tasks
Deterministic Setup
🔎 Similar Papers
No similar papers found.
