Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning

2026-06-25

AI Summary

This work addresses the limited planning capability and poor cross-site generalization commonly observed in small-scale open-source multimodal large language models (MLLMs) for GUI task planning. To overcome these limitations, the authors propose the Planning Experience Exploration and Utilization (PEEU) framework, which autonomously explores interactive environments to collect experience and introduces a novel post-hoc high-level task construction mechanism to synthesize highly aligned training data for fine-tuning compact models. Additionally, they develop the Task Decomposition Hierarchical Analysis Framework (TDHAF), which reveals the critical role of high-level task training in out-of-distribution generalization. Experimental results demonstrate that a 7B-parameter model trained with this approach achieves 30.6% accuracy on real-world benchmarks, substantially outperforming the much larger Qwen2.5-VL-32B model and significantly enhancing both planning and generalization capabilities in small MLLMs.

Abstract

Multimodal web agents can assist humans in operating repetitive GUI tasks, where effective task planning is essential for decomposing complex tasks into executable actions. While small open source MLLMs are cost efficient and privacy preserving compared with commercial large models, they suffer from weak planning and limited cross website generalization. To address these limitations, we introduce the planning experience exploration and utilization (PEEU) method, which autonomously explores environments to discover experiences and utilizes hindsight experience to synthesize strictly aligned, high level training data. To quantitatively analyze the generalization behaviors driving this performance, we propose the task decomposition hierarchical analysis framework (TDHAF) to systematically study compositional generalization across three task granularities: low, middle and high levels. Our analysis reveals that mastering low level atomic skills does not guarantee high level planning competence, while high level task training yields stronger OOD generalization. Experiments on real world benchmarks demonstrate PEEU's superior effectiveness: our 7B model achieves 30.6% accuracy, outperforming the much larger Qwen2.5-VL-32B model. These demonstrate constructing hindsight high level tasks and leveraging experiences is crucial for OOD planning abilities of small MLLMs.

Problem

• task planning
• multimodal web agents
• compositional generalization
• out-of-distribution (OOD) generalization
• GUI automation

Innovation

• PEEU
• hindsight experience
• task planning
• compositional generalization
• multimodal web agents