Towards Scalable Lightweight GUI Agents via Multi-role Orchestration

2026-04-15

AI Summary

This work addresses the limitations of lightweight GUI agents on resource-constrained devices, which typically suffer from restricted capabilities, poor task scalability, and challenges in multi-agent coordination. To overcome these issues, we propose LAMO, a framework that endows compact multimodal large language models (MLLMs) with GUI-specific knowledge and multi-role collaboration skills through role-oriented data synthesis and a two-stage training strategy—combining supervised fine-tuning with perplexity-weighted cross-entropy optimization and reinforcement learning. LAMO supports both single-agent execution and multi-agent orchestration, enabling edge-deployed MLLMs to achieve an unprecedented balance between low-cost deployment and scalable task performance in GUI automation. It also seamlessly integrates plug-and-play compatibility and cooperation with high-level planners. Experiments with LAMO-3B demonstrate its effectiveness and superior performance across both static benchmarks and real-world online GUI workflows.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Autonomous Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) enable digital automation on end-user devices. While scaling both parameters and data has yielded substantial gains, advanced methods still suffer from prohibitive deployment costs on resource-constrained devices. When facing complex in-the-wild scenarios, lightweight GUI agents are bottlenecked by limited capacity and poor task scalability under end-to-end episodic learning, impeding adaptation to multi-agent systems (MAS), while training multiple skill-specific experts remains costly. Can we strike an effective trade-off in this cost-scalability dilemma, enabling lightweight MLLMs to participate in realistic GUI workflows? To address these challenges, we propose the LAMO framework, which endows a lightweight MLLM with GUI-specific knowledge and task scalability, allowing multi-role orchestration to expand its capability boundary for GUI automation. LAMO combines role-oriented data synthesis with a two-stage training recipe: (i) supervised fine-tuning with Perplexity-Weighted Cross-Entropy optimization for knowledge distillation and visual perception enhancement, and (ii) reinforcement learning for role-oriented cooperative exploration. With LAMO, we develop a task-scalable native GUI agent, LAMO-3B, supporting monolithic execution and MAS-style orchestration. When paired with advanced planners as a plug-and-play policy executor, LAMO-3B can continuously benefit from planner advances, enabling a higher performance ceiling. Extensive static and online evaluations validate the effectiveness of our design.

Problem

• lightweight GUI agents
• scalability
• resource-constrained devices
• multi-agent systems
• cost-scalability trade-off

Innovation

• Multi-role Orchestration
• Lightweight MLLM
• GUI Automation
• Role-oriented Training
• Task Scalability