LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control

2026-06-15

AI Summary

Existing agent evaluation benchmarks predominantly focus on virtual software interactions and fail to assess the multimodal interface coordination and feedback-driven parameter tuning required for scientific instrument control. This work introduces the first benchmark specifically designed for this domain, presenting a web-based, extensible, secure, and reproducible simulator suite encompassing eight instrument types and 96 subtasks that fully span the workflow from sample loading to result inspection. The benchmark supports flexible task configuration and execution-based evaluation, integrating vision-language models with a dedicated agent framework. Experimental results demonstrate that while current agents can handle structured GUI subtasks, they struggle significantly with feedback-driven operations and long-horizon workflows, thereby validating the benchmark’s necessity and its capacity to expose critical gaps in agent capabilities.

Abstract

Current computer-use benchmarks primarily focus on software operation tasks in virtualized systems, whereas scientific instrumentation scenarios require coordinated control over complex interfaces, and feedback-driven parameter adjustment. However, directly evaluating agents on physical high-precision instruments is impractical due to high cost, safety risks, limited accessibility, and difficulty in ensuring reproducible evaluation. This motivates the need for a simulated yet realistic testbed that preserves the operational challenges of scientific instruments while enabling scalable and safe benchmarking. To this end, we introduce LabOSBench, a challenging benchmark for multimodal GUI agents built on a suite of web-based scientific-instrument simulators. Operating directly via a browser, LabOSBench avoids resource-heavy OS virtualization while supporting flexible task configuration and execution-based evaluation. Specifically, LabOSBench constructs 96 subtasks across eight instrument simulators, covering workflows from sample loading, alignment, parameter tuning, and data acquisition to result inspection. We evaluate general-purpose vision-language models, specialized GUI agent models, and advanced agentic frameworks at both subtask and end-to-end levels. Our experiments reveal that while existing agents can complete many structured GUI subtasks, they still struggle with feedback-driven operations and long-horizon workflow execution. Overall, LabOSBench provides a reproducible, low-cost testbed for advancing computer-using agents toward scientific-instrument control.

Problem

• scientific instrument control
• computer-use agents
• benchmarking
• simulated testbed
• feedback-driven operation

Innovation

• scientific instrument control
• computer-use agents
• multimodal GUI benchmark
• simulation-based evaluation
• feedback-driven operation