As It Was: Aligning LLM Search Evaluation with Historical User Preferences

2026-07-01

AI Summary

This work addresses the misalignment between existing LLM-as-a-judge approaches and true user preferences in search relevance evaluation, particularly under ambiguous queries and multilingual settings. The authors propose a behavior-anchored LLM evaluator that innovatively structures historical user interaction data into Query-Relevance-Impressions (QRI) cards—lightweight, auditable priors integrated into the LLM’s reasoning process. This approach preserves the model’s semantic understanding while substantially improving alignment with user preferences. Experiments on Spotify music search demonstrate a 5% increase in Spearman correlation and a 91% reduction in disagreement cases. Further evaluation on a five-language human-annotated dataset shows an additional 15% gain in correlation, with results closely matching those of winning models in live A/B tests.

Abstract

Large-scale search systems evolve faster than human quality assurance can scale, especially for long-tail intents and multilingual queries. LLM-as-a-judge approaches provide a scalable alternative for evaluating the relevance of search engine result pages (SERPs), but judgments based solely on semantic similarity or world knowledge can drift from actual user preferences, particularly for ambiguous queries. We introduce a behavior-grounded LLM judge that augments each SERP item with a lightweight and auditable behavioral prior in the form of a Query-Relevance-Impressions (QRI) card. Each card summarizes how users have historically interacted with similar queries and results, providing compact empirical evidence that the judge can cite to resolve ambiguity and make more consistent relevance judgments while still relying on semantic reasoning. In a large-scale music search evaluation at Spotify, using relevance estimates derived from historical user interactions across 6,000 recomposed SERPs, the behavior-grounded judge achieves stronger alignment with user preferences, improving Spearman rank correlation by approximately 5% overall and yielding a 91% relative improvement on disagreement cases. On a multilingual human-judged dataset spanning five languages, grounding further increases correlation with human relevance judgments by 15%. Importantly, when evaluated against outcomes from a live A/B test, the grounded judge shows consistently higher alignment with the observed winning model. While absolute alignment remains moderate, these findings demonstrate that lightweight behavioral grounding can improve the reliability and practical usefulness of LLM-based evaluation in real-world search systems.

Problem

• LLM-as-a-judge
• search evaluation
• user preferences
• relevance judgment
• behavioral grounding

Innovation

• behavior-grounded LLM
• search evaluation
• user preferences
• QRI card
• relevance judgment