ColorAgent: Building A Robust, Personalized, and Interactive OS Agent

2025-10-22

AI Summary

This work addresses the challenges of achieving long-horizon robust interaction, personalized response, and proactive collaboration in operating system (OS) intelligent agents. We propose a self-evolving training framework based on a multi-agent architecture, integrating large language models, progressive reinforcement learning, and environment-driven self-evolution mechanisms. The framework enables end-to-end training of system-level OS agents within the AndroidWorld and AndroidLab environments. To our knowledge, it is the first to realize an OS-level intelligent agent capable of intent understanding, persistent state tracking, and proactive task coordination. Experiments demonstrate state-of-the-art performance, achieving 77.2% and 50.7% task success rates on the AndroidWorld and AndroidLab benchmarks, respectively—substantially surpassing prior approaches. The code is publicly released to advance research in OS-level embodied intelligence.

Technology Category

• Multiagent Systems: Multiagent Learning
• Intelligent Robots: Multi-Robot Systems
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

With the advancements in hardware, software, and large language model technologies, the interaction between humans and operating systems has evolved from the command-line interface to the rapidly emerging AI agent interactions. Building an operating system (OS) agent capable of executing user instructions and faithfully following user desires is becoming a reality. In this technical report, we present ColorAgent, an OS agent designed to engage in long-horizon, robust interactions with the environment while also enabling personalized and proactive user interaction. To enable long-horizon interactions with the environment, we enhance the model's capabilities through step-wise reinforcement learning and self-evolving training, while also developing a tailored multi-agent framework that ensures generality, consistency, and robustness. In terms of user interaction, we explore personalized user intent recognition and proactive engagement, positioning the OS agent not merely as an automation tool but as a warm, collaborative partner. We evaluate ColorAgent on the AndroidWorld and AndroidLab benchmarks, achieving success rates of 77.2% and 50.7%, respectively, establishing a new state of the art. Nonetheless, we note that current benchmarks are insufficient for a comprehensive evaluation of OS agents and propose further exploring directions in future work, particularly in the areas of evaluation paradigms, agent collaboration, and security. Our code is available at https://github.com/MadeAgents/mobile-use.

Problem

• Develops robust OS agent for long-horizon environmental interactions
• Enables personalized user intent recognition and proactive engagement
• Builds interactive AI agent replacing traditional command-line interfaces

Innovation

• Uses step-wise reinforcement learning for training
• Develops tailored multi-agent framework for robustness
• Implements personalized user intent recognition proactively