Diagnosing and Mitigating Retrieval Bottlenecks in LLM-Based Cold-Start Recommendation

2026-06-29

AI Summary

This work addresses the limitations of large language model (LLM)-based reranking in cold-start recommendation, where performance is constrained by low coverage of new items during retrieval. The study presents the first decoupled evaluation of retrieval coverage and reranking quality, introducing a cross-domain cold-start benchmark spanning five domains. To improve coverage, the authors propose LHF, a learnable hybrid fusion method that combines multiple retrievers via a validation-set-trained fusion layer. LHF consistently outperforms individual retrievers across all domains, recovering 17–61% of ideal coverage in content-rich settings. However, end-to-end experiments reveal a structural mismatch between retrieval and reranking: current prompt-based LLM rerankers often fail to effectively leverage the enhanced retrieval results, frequently diminishing the gains achieved by LHF.

Abstract

Large language models (LLMs) are increasingly used as rerankers in recommender systems, with the expectation that semantic understanding will help in cold-start and long-tail regimes. We test this assumption with a five-domain benchmark that explicitly separates reranking quality from retrieval coverage. In a positive-controlled regime where the gold item is guaranteed present, calibrated LLM rerankers fail to consistently outperform strong collaborative and content baselines under natural traffic, and within-family scaling from Qwen3-8B to Qwen3-32B narrows but does not close the gap on most domains. In a retrieval-realistic regime where the gold item is not injected, the bottleneck is more severe: standard single retrievers place the gold item in a 200-item pool only 4.6-22.9% of the time, largely because 32-91% of cold-start targets are brand-new items with no training interactions. We introduce LHF, a validation-trained learned hybrid fusion layer over a multi-retriever union pool, as a retrieval-side realizability baseline. LHF is the only combiner we test that beats every single retriever on all five domains and recovers 17-61% of oracle coverage headroom on content-rich domains, but only 5-7% on collaboratively strong domains. End-to-end experiments reveal the remaining mismatch: learned non-LLM ranking exploits the LHF pool, while prompt-level LLM reranking often degrades it. LLMs exhibit pockets of semantic cold-start advantage, especially in text-rich domains when the item is already present, but this advantage is largely unreachable in current retrieve-then-rerank pipelines. We release the benchmark protocol, splits, prompts, evaluation tooling, and archived reproducibility artifacts: data at https://doi.org/10.5281/zenodo.20991039 and code at https://doi.org/10.5281/zenodo.20993306.

Problem

• cold-start recommendation
• retrieval bottleneck
• large language models
• reranking
• item coverage

Innovation

• retrieval bottleneck
• cold-start recommendation
• learned hybrid fusion
• LLM reranking
• multi-retriever fusion