Learning from Online User Feedback for Shopping Agents

2026-08-11

AI Summary

This work addresses the challenge that existing shopping agents predominantly rely on offline training signals and struggle to effectively leverage sparse, heterogeneous, and noisy online natural feedback from user interactions. To overcome this limitation, the authors propose LOFA, a novel framework that, for the first time, extracts fine-grained token-level supervision signals directly from real user interaction logs without requiring any human annotation. LOFA integrates reinforcement learning grounded in verifiable purchase behaviors with feedback-aware on-policy policy distillation, transforming user instructions into reliable learning signals while jointly optimizing behavioral patterns and personalized preferences. Experiments on real-world e-commerce logs demonstrate that LOFA significantly outperforms strong baselines, achieving substantial improvements in recommendation quality, response helpfulness, and alignment with user satisfaction.

Abstract

Large language model-based shopping agents are increasingly deployed in real-world e-commerce platforms, generating massive amounts of user interaction logs that provide valuable supervision for improving these agents. However, existing approaches primarily rely on offline training signals, such as user-item interactions or synthetic preference data, while largely overlooking the rich supervision contained in users' natural conversational feedback. Moreover, the available online feedback is heterogeneous, sparse, and noisy, making it difficult to transform into reliable learning signals automatically. To address these challenges, we propose LOFA, a framework that enables shopping agents to learn directly from real online interaction logs without human annotation. LOFA combines reinforcement learning over verifiable purchase outcomes with feedback-aware on-policy distillation, which identifies users'in-dialogue directives and converts them into dense token-level supervision. These complementary objectives capture both collaborative behavioral patterns and user-specific preferences. Extensive experiments on real-world e-commerce logs demonstrate that LOFA consistently improves recommendation quality, response helpfulness, and user-satisfaction alignment over strong baselines, highlighting the effectiveness of learning shopping agents from real online user feedback.

Problem

• online user feedback
• shopping agents
• heterogeneous feedback
• noisy supervision
• conversational feedback

Innovation

• online user feedback
• shopping agents
• reinforcement learning
• on-policy distillation
• token-level supervision