AGPO: Asymmetric Group Policy Optimization for Verifiable Reasoning and Search Ads Relevance at JD

2026-05-07

AI Summary

This work addresses the tendency of existing reinforcement learning methods to narrow the reasoning capability boundary of large language models through over-optimization, thereby diminishing their ability to explore diverse correct reasoning paths. To mitigate this, the paper proposes Asymmetric Group Policy Optimization (AGPO), which uniquely integrates a negative-dominant policy with a group advantage mechanism: the former suppresses erroneous reasoning trajectories, while the latter enhances learning of rare yet valid paths via intra-group variance. Built upon the RLVR reward framework and asymmetric policy gradients, AGPO achieves state-of-the-art accuracy across five mathematical reasoning benchmarks and substantially improves large-scale pass@$k$ performance. Furthermore, when applied to JD.com’s search ad relevance annotation task, AGPO effectively boosts the performance of downstream student models.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated notable success in enhancing the reasoning performance of large language models (LLMs). However, recent studies reveal that while current RLVR methods improve sampling efficiency towards correct paths, they do not elicit fundamentally new reasoning patterns. Instead, the reasoning capability boundary of trained models often narrows compared to their base models, with base models achieving higher coverage at large sample sizes. In this work, we propose Asymmetric Group Policy Optimization (AGPO) to counteract this boundary shrinkage. AGPO adopts a negative-dominant reinforcement strategy to suppress incorrect reasoning paths, maintaining the base model's exploration capacity. For positive reinforcement, AGPO adopts a group advantage mechanism, which scales positive updates based on intra-group variance, allowing the model to focus on rare correct paths while suppressing updates from trivial paths. Our experiments on five mathematical benchmarks demonstrate that AGPO achieves state-of-the-art accuracy while consistently improving pass@$k$ performance at scale. In a large-scale industrial application for search ads relevance optimization, AGPO effectively enhances the quality of the data annotation, leading to substantial performance gains in downstream student models.

Problem

• Reinforcement Learning with Verifiable Rewards
• reasoning capability boundary
• large language models
• sampling efficiency
• coverage

Innovation

• Asymmetric Group Policy Optimization
• Reinforcement Learning with Verifiable Rewards
• reasoning boundary preservation
• group advantage mechanism
• search ads relevance