OpenComputer: Verifiable Software Worlds for Computer-Use Agents

2026-05-19

AI Summary

Current agent evaluation lacks verifiable, structured desktop software environments, leading to significant discrepancies with human judgment—particularly in fine-grained state assessment. This work proposes the first validator-driven framework encompassing 33 desktop applications, establishing a realistic and reproducible interactive environment through application-specific state validators, an execution-feedback-driven self-evolving validation layer, a machine-checkable task generation pipeline, and an auditable scoring system supporting partial credit. Evaluated on 1,000 tasks, the approach substantially improves alignment with human judgments compared to LLM-as-judge baselines. Experimental results further reveal that even state-of-the-art agents struggle to complete tasks end-to-end, with open-source models exhibiting markedly degraded performance, thereby highlighting a critical gap in robust automation capabilities.

Abstract

We present OpenComputer, a verifier-grounded framework for constructing verifiable software worlds for computer-use agents. OpenComputer integrates four components: (1) app-specific state verifiers that expose structured inspection endpoints over real applications, (2) a self-evolving verification layer that improves verifier reliability using execution-grounded feedback, (3) a task-generation pipeline that synthesizes realistic and machine-checkable desktop tasks, and (4) an evaluation harness that records full trajectories and computes auditable partial-credit rewards. In its current form, OpenComputer covers 33 desktop applications and 1,000 finalized tasks spanning browsers, office tools, creative software, development environments, file managers, and communication applications. Experiments show that OpenComputer's hard-coded verifiers align more closely with human adjudication than LLM-as-judge evaluation, especially when success depends on fine-grained application state. Frontier agents struggle with end-to-end completion despite partial progress, and open-source models exhibit sharp drops from their OSWorld-Verified scores, exposing a persistent gap in robust computer automation.

Problem

• computer-use agents
• verifiable software worlds
• task evaluation
• application state verification
• desktop automation

Innovation

• verifiable software worlds
• state verifiers
• self-evolving verification
• task generation pipeline
• computer-use agents