Problem
Research questions and friction points this paper is trying to address.
Graphical User Interface
Standard Operating Procedures
Domain-specific Knowledge
Task Verification
OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations
🤖 AI Summary
研究解决了GUI代理执行专业标准操作程序（SOP）的难题，通过OmegaUse-SOP系统将人类演示转化为可复用技能，提高代理在特定领域软件中的可靠性。
📝 Abstract
Large language models (LLMs) are increasingly evolving from conversational assistants into agents capable of operating external digital environments. Graphical user interface (GUI) agents play an important role in this transition, as many real-world workflows remain accessible only through user-facing software interfaces. However, despite recent progress on general computer-use benchmarks, domain-specific professional standard operating procedures (SOPs) remain challenging for GUI agents because they often involve implicit domain knowledge, software-specific conventions, and task-level verification requirements. We introduce OmegaUse-SOP, a human-in-the-loop SOP Engineering system for transforming human demonstrations of professional computer use into reusable SOP skills for GUI agents. Analogous to prompt engineering, SOP Engineering iteratively refines demonstrations, execution rules, and domain knowledge to convert professional SOPs into reusable GUI-agent skills. OmegaUse-SOP consists of four modules: Observe, Reason, Configure, and Execute. Together, these modules record expert operations as multimodal GUI traces, abstract low-level events into semantic step-level instructions, incorporate domain rules and task-specific parameters, and execute the resulting skills in live GUI environments through step-wise grounding, action generation, and verification. To demonstrate its effectiveness, we collaborate with a power-sector client and test OmegaUse-SOP on photovoltaic simulation workflows in PVsyst 7.2. The results suggest that OmegaUse-SOP can improve GUI-agent reliability on professional SOP tasks, highlighting a practical path toward deploying GUI agents in domain-specific professional software environments.
Innovation
Methods, ideas, or system contributions that make the work stand out.
SOP Engineering
Human Demonstrations
GUI Agents
Professional Software Environments
Step-wise Execution
🔎 Similar Papers
No similar papers found.
