User Preference Induction with LLMs for Offline Top-N Recommendation Evaluation

2026-07-13

AI Summary

Offline Top-N recommendation evaluation is prone to distortion due to data sparsity and popularity bias, as it conventionally treats unobserved items as irrelevant. To address this limitation, this work proposes the first approach that leverages large language models (LLMs) to construct textual user preference profiles and employs these profiles to infer item relevance for candidate recommendations. By expanding the set of relevance labels beyond observed interactions, the method mitigates evaluation bias induced by sparse feedback, thereby enhancing the completeness, fairness, and robustness of offline evaluation protocols.

Abstract

Offline evaluation is the standard methodology for comparing top-N recommender systems, yet it relies on incomplete relevance information. In most benchmark datasets, only a small subset of user--item preferences is observed, and unjudged items are commonly treated as non-relevant. This missing-as-negative assumption can bias evaluation, penalize plausible recommendations with no recorded feedback, and favour algorithms that concentrate on popular or highly exposed items. We propose an LLM-based framework to expand relevance judgements for offline recommender evaluation. Our approach uses large language models in two complementary roles. First, a preference induction stage summarizes each user's historical interactions into a textual profile that captures their tastes and interests. Second, conditioned on this profile, an LLM acts as a relevance judge for candidate recommended items that lack observed labels in the original test data. To make this process tractable and evaluation-focused, we apply judgement expansion to a pooled candidate set built from the top-ranked outputs of multiple recommenders. The resulting enriched judgements provide additional relevance evidence for previously unobserved user--item pairs, enabling ranking metrics to be computed on a more complete basis. Experimental results show that this approach is a promising strategy for improving the robustness of offline top-N evaluation and mitigating the popularity-sensitive distortions caused by sparse feedback.

Problem

• offline evaluation
• top-N recommendation
• missing-as-negative
• relevance judgment
• evaluation bias

Innovation

• LLM-based preference induction
• offline recommendation evaluation
• relevance judgment expansion
• top-N recommendation
• popularity bias mitigation