HATS: Hardness-Aware Trajectory Synthesis for GUI Agents

2026-03-12

AI Summary

Existing methods for GUI agent trajectory synthesis often overlook semantically ambiguous actions, leading to misalignment between instructions and execution and limiting generalization. This work proposes a hardness-aware trajectory synthesis framework that, for the first time, formalizes the semantic ambiguity of actions as “hardness” and establishes a closed-loop mechanism of exploration and refinement. By leveraging hardness-driven active exploration to prioritize high-information interactions and integrating alignment-guided iterative refinement, the framework dynamically enhances trajectory quality and instruction-execution consistency. Built upon large vision-language models, the approach significantly outperforms current state-of-the-art baselines across multiple GUI benchmarks, effectively improving agent robustness and generalization.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Graphical user interface (GUI) agents powered by large vision-language models (VLMs) have shown remarkable potential in automating digital tasks, highlighting the need for high-quality trajectory data to support effective agent training. Yet existing trajectory synthesis pipelines often yield agents that fail to generalize beyond simple interactions. We identify this limitation as stemming from the neglect of semantically ambiguous actions, whose meanings are context-dependent, sequentially dependent, or visually ambiguous. Such actions are crucial for real-world robustness but are under-represented and poorly processed in current datasets, leading to semantic misalignment between task instructions and execution. To address these issues, we propose HATS, a Hardness-Aware Trajectory Synthesis framework designed to mitigate the impact of semantic ambiguity. We define hardness as the degree of semantic ambiguity associated with an action and develop two complementary modules: (1) hardness-driven exploration, which guides data collection toward ambiguous yet informative interactions, and (2) alignment-guided refinement, which iteratively validates and repairs instruction-execution alignment. The two modules operate in a closed loop: exploration supplies refinement with challenging trajectories, while refinement feedback updates the hardness signal to guide future exploration. Extensive experiments show that agents trained with HATS consistently outperform state-of-the-art baselines across benchmark GUI environments.

Problem

• semantic ambiguity
• trajectory synthesis
• GUI agents
• instruction-execution alignment
• generalization

Innovation

• Hardness-Aware
• Trajectory Synthesis
• Semantic Ambiguity
• Alignment-Guided Refinement
• GUI Agents