Don't Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target

2026-05-17

AI Summary

This work addresses the issue of policy update deviation in generative large language model–based recommender systems under continual learning, where only biased contextual bandit feedback—characterized by reliable positive signals and ambiguous non-responses due to exposure bias—is available. To mitigate this, the paper proposes the Anchored Bandit Policy Optimization (ABPO) framework, which innovatively treats historically exposed items as fixed anchors within a group-wise relative policy optimization scheme. ABPO jointly alleviates exposure bias and negative signal ambiguity by integrating inverse propensity score weighting with an asymmetric feedback reliability model grounded in the recommender’s output confidence. Experiments across five domains from Amazon Reviews and MovieLens demonstrate that ABPO significantly improves recommendation accuracy while effectively reducing bias induced by historical deployment policies.

Abstract

Generative LLM-based recommenders (LLM-Rec) require continual post-deployment updates, yet deployment logs provide only policy-shaped contextual bandit feedback: outcomes are observed solely for items exposed by a prior serving policy, inducing exposure bias and yielding partial, asymmetric signals consisting of relatively reliable positive responses and ambiguous no-responses. We propose an Anchored Bandit Policy Optimization (ABPO) framework for continual LLM-Rec updates that combines group-relative policy optimization (GRPO) with explicit treatment of exposure bias and feedback ambiguity. Specifically, we insert the exposed recommendation as a logged anchor into each GRPO rollout group, so that group-relative normalization is calibrated against the action actually exposed by the prior policy rather than against newly sampled rollouts alone. Because both positive- and no-responses are observed only through prior-policy exposure, we apply self-normalized inverse propensity scoring to the fixed anchor for both feedback types to correct for policy mismatch. At the same time, we treat the two feedback types asymmetrically in reliability: positive responses provide relatively direct endorsement signals, whereas no-responses remain ambiguous because they may reflect either true disinterest or unobserved external factors. To avoid overly aggressive updates from ambiguous no-responses, we temper their penalties with self-certainty, using the model's output-token confidence as a verifier-free reliability signal. Across five domains from Amazon Reviews and MovieLens, our method yields consistent post-update gains in recommendation accuracy while mitigating prior-policy-induced exposure bias more effectively than prior baselines.

Problem

• exposure bias
• bandit feedback
• LLM-based recommender
• continual learning
• feedback ambiguity

Innovation

• Anchored Bandit Policy Optimization
• exposure bias correction
• self-normalized inverse propensity scoring
• asymmetric feedback reliability
• continual LLM-based recommendation