Problem
Research questions and friction points this paper is trying to address.
cross-platform GUI interaction
self-evolution mechanism
task automation
user interaction experience
long-horizon task
KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill
🤖 AI Summary
This work addresses the limitations of existing GUI agents in cross-platform compatibility and continuous self-evolution. It proposes a “Deep Cognition, Precise Execution” paradigm through a Know-Route-Act-Reflect framework that integrates user interaction experience with task-specific knowledge to enable long-horizon task decomposition, cross-platform GUI manipulation, and feedback-driven self-optimization. Key innovations include an attributable memory system and a plugin-based sub-agent architecture with a self-evolving skill library, facilitating seamless migration and rapid integration across multiple platforms—including Android, iOS, HarmonyOS, and Windows. Evaluated on the MobileWorld benchmark, the approach achieves a 64.1% success rate, substantially outperforming current methods. Furthermore, its memory and skill mechanisms are transferable across different foundation models, yielding an 8.5% performance improvement.
📝 Abstract
OpenClaw has emerged as a leading agent framework for complex task automation, yet it faces insufficient cross-platform GUI interaction support and a well-built self-evolution mechanism. These flaws limit its adaptation to diverse device ecosystems and prevent performance improvements through continuous learning from execution experience. To resolve these issues, we propose the Know Deeply, Act Perfectly paradigm for personal assistants, which holds that accumulated user interaction and task-running experience directly improve execution accuracy and efficiency, unifying cognitive comprehension and operational execution. Based on this paradigm, we introduce KnowAct-GUIClaw, a novel Know-Route-Act-Reflect framework designed to address OpenClaw's GUI manipulation deficits and break through its cross-platform and recursive self-improvement constraints. First, the host agent leverages accumulated interaction experience and task-relevant knowledge for long-horizon task decomposition and allocation (Know). Second, a pluggable GUI subagent with an experience-attributable memory system (Know) and self-evolving skill library (Act), enabling seamless cross-platform migration and fast-path integration. Especially, this framework continuously stores user profiles and feedback to improve the accuracy of task decomposition and tool calls. Extensive experiments across Android, iOS, HarmonyOS and Windows show that KnowAct-GUIClaw achieves superior efficiency, accuracy and cross-platform adaptability. Especially, the GUIClaw with open-source Kimi-2.6 models achieves the best performance (64.1%) on the long-horizon MobileWorld benchmark, beating all agentical frameworks and closed-source agentical models, e.g., Seed-2.0-Pro and GPT-5.5. Additionally, the knowledgeable memory and execution skills supported by our framework are transferable across diverse base models, improving by 8.5% with Kimi-2.6.
Innovation
Methods, ideas, or system contributions that make the work stand out.
self-evolving memory
cross-platform GUI agent
Know-Route-Act-Reflect framework
experience-attributable memory
long-horizon task automation
🔎 Similar Papers
No similar papers found.
