CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications

2026-08-11

AI Summary

Existing mobile GUI agents exhibit limited generalization when encountering unseen applications during deployment. This work proposes CoAdapt-GUI, a novel framework that jointly adapts structured workflow context and policy at test time—a first in the field. By leveraging the agent’s interaction trajectories and rewards within the target application, CoAdapt-GUI disentangles transferable procedural knowledge from source-specific interface details and uses constraint-guided workflow context to steer policy optimization. The approach integrates test-time adaptation (TTA), LoRA adapters, a frozen vision-language model, and group-relative optimization aligned with task context. Evaluated on AndroidWorld-Generalization, it achieves a success rate of 45.0%, outperforming a policy-only adaptation baseline (37.5%), and significantly boosts performance on AndroidWorld Plus from 38.6% to 52.9%.

Abstract

Mobile GUI agents remain brittle when deployed to applications absent from source training. We study novel-app generalization under a limited target interaction budget and without target demonstrations. We introduce CoAdapt-GUI, a test-time adaptation (TTA) framework that jointly adapts structured workflow context and policy from the agent's own target-app rollouts and rewards. The workflow context retains transferable procedures, failure modes, and verification rules while excluding app-bound source details. This separation allows reusable workflow knowledge to guide adaptation without transferring source-interface state. For policy adaptation, task-context-matched group-relative optimization updates a LoRA adapter on a frozen vision-language model. Across two unseen-app evaluations, CoAdapt-GUI reaches 45.0% on AndroidWorld-Generalization, compared with 37.5% for the reported Policy-Only TTA baseline, and raises AndroidWorld Plus performance from 38.6% to 52.9%. These results show that transfer-constrained workflow context provides substantial gains and that joint policy adaptation further improves held-out performance.

Problem

• GUI agents
• novel-app generalization
• test-time adaptation
• workflow context
• policy adaptation

Innovation

• test-time adaptation
• workflow context
• policy adaptation
• GUI agents
• generalization