Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders

2026-04-24

AI Summary

This work addresses the misalignment between optimizing area under the ROC curve (AUC) and the Top-K recommendation objective in existing reinforcement learning–based large language model (LLM) recommender systems. The study reveals, for the first time, that hard negative samples bias optimization toward local AUC regions, thereby degrading Top-K performance. To bridge this gap, the authors propose windowed partial AUC (WPAUC) as a more suitable optimization target aligned with Top-K metrics, along with a Threshold-Adjusted Window reWeighting (TAWin) algorithm to efficiently optimize WPAUC. Integrated within the Group Relative Policy Optimization framework, the proposed approach achieves state-of-the-art Top-K recommendation performance across four real-world datasets.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search
• Data Mining & Knowledge Management: Recommender Systems

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Reinforcement learning (RL) effectively optimizes Large Language Model (LLM)-based recommenders by contrasting positive and negative items. Empirically, training with beam-search negatives consistently outperforms random negatives, yet the mechanism is not well understood. We address this gap by analyzing the induced optimization objective and show that: (i) Under binary reward feedback, optimizing LLM recommenders with Group Relative Policy Optimization (GRPO) is theoretically equivalent to maximizing the Area Under the ROC Curve (AUC), which is often misaligned with Top-$K$ recommendation; and (ii) Replacing random negatives with beam-search negatives reshapes the objective toward partial AUC, improving alignment with Top-$K$ metrics. Motivated by this perspective, we introduce Windowed Partial AUC (WPAUC), which constrains the false positive rate (FPR) to a window [$α,α+d$] to more directly align with Top-$K$ metrics. We further propose an efficient Threshold-Adjusted Windowed reweighting (TAWin) RL method for its optimization, enabling explicit control over the targeted Top-$K$ performance. Experiments on four real-world datasets validate the theory and deliver consistent state-of-the-art performance.

Problem

• Top-K recommendation
• partial AUC
• reinforcement learning
• LLM-based recommenders
• objective misalignment

Innovation

• Windowed Partial AUC
• Top-K Recommendation
• Hard Negatives
• Reinforcement Learning
• LLM-based Recommenders