TGPO: Tree-Guided Preference Optimization for Robust Web Agent Reinforcement Learning

2025-09-17

AI Summary

Addressing the core challenges of misaligned credit assignment, sparse rewards, and high annotation costs in web-agent reinforcement learning, this paper proposes Tree-Guided Preference Optimization (TGPO). TGPO models agent trajectories as trees and merges semantically equivalent states to resolve label conflicts. It introduces a Process Reward Model (PRM) that jointly evaluates subgoal progress, detects redundant actions, and verifies action validity—enabling fine-grained, automated dense reward generation. A dynamic weighting mechanism further prioritizes critical decision points during training. Experiments on Online-Mind2Web and C-WebShop demonstrate that TGPO significantly improves task success rates, yields more concise execution paths, reduces redundant steps by 37.2%, and achieves superior training efficiency and policy robustness compared to state-of-the-art methods.

Technology Category

• Machine Learning: Reinforcement Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

With the rapid advancement of large language models and vision-language models, employing large models as Web Agents has become essential for automated web interaction. However, training Web Agents with reinforcement learning faces critical challenges including credit assignment misallocation, prohibitively high annotation costs, and reward sparsity. To address these issues, we propose Tree-Guided Preference Optimization (TGPO), an offline reinforcement learning framework that proposes a tree-structured trajectory representation merging semantically identical states across trajectories to eliminate label conflicts. Our framework incorporates a Process Reward Model that automatically generates fine-grained rewards through subgoal progress, redundancy detection, and action verification. Additionally, a dynamic weighting mechanism prioritizes high-impact decision points during training. Experiments on Online-Mind2Web and our self-constructed C-WebShop datasets demonstrate that TGPO significantly outperforms existing methods, achieving higher success rates with fewer redundant steps.

Problem

• Addresses credit assignment misallocation in web agents
• Reduces high annotation costs through automated rewards
• Solves reward sparsity with fine-grained progress tracking

Innovation

• Tree-structured trajectory representation merging states
• Process Reward Model generating fine-grained rewards
• Dynamic weighting prioritizing high-impact decisions