LLM-as-a-Judge for Reliable and Explainable Offline Evaluation in Top-K Recommendation

2026-06-22

AI Summary

Traditional Top-K offline recommendation evaluation relies on biased and incomplete user behavior ID matching, suffering from limited reliability and interpretability. This work proposes a novel evaluation framework grounded in large language models (LLMs): it constructs semantic preference proxies from users’ textual interactions, enables flexible matching of recommendations within a semantic space, and adopts a “reason-then-score” paradigm to generate relevance judgments accompanied by justifications. By replacing rigid ID-based matching with semantic alignment, the approach substantially enhances evaluation reliability while leveraging the LLM’s reasoning capabilities to yield interpretable scores. Experimental results demonstrate that the proposed framework consistently outperforms existing methods in terms of reliability, interpretability, and robustness, effectively identifying both successful and failed recommendations with well-reasoned explanations.

Abstract

Recommendation evaluation plays a crucial role in guiding the refinement and deployment of recommender systems. Most existing trials rely on offline evaluation using Top-K metrics computed over holdout user behaviors. However, we identify two fundamental limitations that undermine their ability to deliver reliable and explainable evaluations. Regarding reliability, offline evaluation treats observed user feedback as a proxy of true preferences and enforces rigid ID matching between the proxy and recommendation. In practice, feedback collections are inherently shaped by incomplete and biased item exposure, leading to distorted and unreliable assessments. Regarding explainability, Top-K metrics only establish numerical scores without offering meaningful insights to support them, thereby reinforcing the black-box nature of offline evaluation. In this paper, we propose a reliable and explainable LLM-as-a-Judge framework for offline recommendation evaluation. To enhance reliability, we introduce a semantic proxy from user textual behaviors to represent their true preferences. This proxy allows for more flexible matching between preferences and recommendations in the semantic space, rather than depending on the holdout feedback. To ensure explainability, the LLM Judge adopts a reasoning-then-scoring process to generate relevance judgments along with explicit rationale. Finally, we aggregate the individual scores into global Top-K metrics to quantify overall recommendation quality, and provide justification for each preference hit or miss. Extensive experiments demonstrate that the LLM Judge achieves solid reliability, explainability, and robustness in evaluation.

Problem

• offline evaluation
• reliability
• explainability
• Top-K recommendation
• LLM-as-a-Judge

Innovation

• LLM-as-a-Judge
• semantic proxy
• explainable evaluation
• offline recommendation
• top-k metrics