ThinkQE: Query Expansion via an Evolving Thinking Process

2025-06-10

AI Summary

Existing LLM-based query expansion methods improve retrieval performance and generalization but often produce semantically narrow, monolithic expansions, limiting result diversity. This paper proposes “Reasoning-Augmented Expansion”: a framework that employs chain-of-thought prompting to simulate multi-step semantic reasoning and integrates retrieval-feedback-driven zero-shot iterative rewriting at inference time, dynamically balancing breadth exploration and precision convergence without additional training. Its core innovation lies in embedding retrieval interaction directly into the reasoning process, thereby enabling deeper semantic expansion and enhanced diversity. Evaluated on standard benchmarks—including DL19, DL20, and BRIGHT—the method consistently outperforms training-intensive dense retrievers and re-rankers, achieving significant gains in nDCG@10 and MAP.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Effective query expansion for web search benefits from promoting both exploration and result diversity to capture multiple interpretations and facets of a query. While recent LLM-based methods have improved retrieval performance and demonstrate strong domain generalization without additional training, they often generate narrowly focused expansions that overlook these desiderata. We propose ThinkQE, a test-time query expansion framework addressing this limitation through two key components: a thinking-based expansion process that encourages deeper and comprehensive semantic exploration, and a corpus-interaction strategy that iteratively refines expansions using retrieval feedback from the corpus. Experiments on diverse web search benchmarks (DL19, DL20, and BRIGHT) show ThinkQE consistently outperforms prior approaches, including training-intensive dense retrievers and rerankers.

Problem

• Improves query expansion for diverse web search results
• Addresses narrow focus in LLM-based expansion methods
• Enhances semantic exploration via iterative corpus feedback

Innovation

• Thinking-based expansion for deeper semantic exploration
• Corpus-interaction strategy for iterative refinement
• Outperforms training-intensive retrievers and rerankers