LegacyWorld: Atomicity-Aware Evaluation of GUI Agents for Legacy Workflows

2026-08-14

AI Summary

This study addresses the challenges of persistent side effects caused by agent failures and the lack of rigorous evaluation in GUI automation for legacy systems. We propose an atomicity-aware evaluation framework that incorporates an expert benchmark equipped with state validators, integrating safe failure mechanisms into acceptance criteria to effectively distinguish between task completion, safe rollback, and non-atomic side effects. This research establishes workflow capture and atomicity testing as fundamental requirements for AI-driven automation, validating the necessity of this paradigm for safeguarding mission-critical enterprise operations. Consequently, this work bridges a significant gap in existing evaluation methodologies by explicitly accounting for operational atomicity and system safety within automated environments.

Abstract

Legacy and legacy-like enterprise systems often remain difficult to modernize because critical workflows expose limited programmable interfaces and still require manual GUI interaction. This paper reports a pre-deployment evaluation study motivated by the development of legacy-use, an industry-oriented framework for automating such workflows with multimodal LLM agents. During framework development, domain experts helped identify stateful workflows where successful demos are not sufficient: a failed agent run may still leave persistent invalid changes in business or healthcare records. We therefore evaluate computer-use agents using atomicity: a run should either complete the intended workflow correctly or fail without unintended persistent side effects. We construct a domain-expert-informed benchmark of 28 Windows GUI workflows, each specified with an initial state, goal state, and task-specific validator. We compare expert-crafted prompts with prompts generated from screen recordings of expert golden-path executions. Across six hosted computer-use agents, our results show that useful completion, safe failure, and non-atomic side effects are distinct operational profiles. We conclude that workflow capture, state validators, and atomicity-aware acceptance tests should be first-class requirements for AI-based legacy workflow automation.

Problem

• GUI Agents
• Legacy Workflows
• Atomicity
• Side Effects
• Evaluation

Innovation

• Atomicity-Aware Evaluation
• GUI Agents
• Legacy Workflows
• State Validators
• Benchmark