On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stability

2026-04-17

AI Summary

This study addresses the lack of systematic evaluation of generalization and stability in large language model (LLM)-based dense retrievers. It presents the first comprehensive analysis of the “specialization tax” phenomenon in LLM retrievers, integrating linear mixed-effects modeling, adversarial attacks, query perturbations, and embedding geometry analysis. The authors demonstrate that geometric properties of the embedding space effectively predict lexical stability and reveal a positive correlation between model scale and robustness. Experimental results show that instruction-tuned models exhibit stronger generalization capabilities, whereas inference-optimized models suffer from limited generalization. Furthermore, LLM retrievers prove robust to spelling errors and corpus poisoning but remain vulnerable to semantic perturbations such as synonym substitution.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Decoder-only large language models (LLMs) are increasingly replacing BERT-style architectures as the backbone for dense retrieval, achieving substantial performance gains and broad adoption. However, the robustness of these LLM-based retrievers remains underexplored. In this paper, we present the first systematic study of the robustness of state-of-the-art open-source LLM-based dense retrievers from two complementary perspectives: generalizability and stability. For generalizability, we evaluate retrieval effectiveness across four benchmarks spanning 30 datasets, using linear mixed-effects models to estimate marginal mean performance and disentangle intrinsic model capability from dataset heterogeneity. Our analysis reveals that while instruction-tuned models generally excel, those optimized for complex reasoning often suffer a ``specialization tax,'' exhibiting limited generalizability in broader contexts. For stability, we assess model resilience against both unintentional query variations~(e.g., paraphrasing, typos) and malicious adversarial attacks~(e.g., corpus poisoning). We find that LLM-based retrievers show improved robustness against typos and corpus poisoning compared to encoder-only baselines, yet remain vulnerable to semantic perturbations like synonymizing. Further analysis shows that embedding geometry (e.g., angular uniformity) provides predictive signals for lexical stability and suggests that scaling model size generally improves robustness. These findings inform future robustness-aware retriever design and principled benchmarking. Our code is publicly available at https://github.com/liyongkang123/Robust_LLM_Retriever_Eval.

Problem

• robustness
• dense retrieval
• large language models
• generalizability
• stability

Innovation

• LLM-based dense retrieval
• robustness evaluation
• generalizability
• stability
• embedding geometry