Demographically-Inspired Query Variants Using an LLM

2025-08-25

AI Summary

This work addresses the lack of user diversity representation in information retrieval test collections. We propose a user persona–driven query diversification method that leverages large language models to generate semantically consistent query variants reflecting heterogeneous linguistic proficiency and domain expertise—thereby simulating real users’ divergent information needs and expression styles. User attributes (e.g., language competence, professional background) are explicitly encoded as constraints during query generation, enabling construction of multi-perspective, representative test sets. Empirical evaluation demonstrates that the generated variants induce statistically significant shifts in system ranking outputs, uncovering previously masked inter-group performance disparities—particularly across user subpopulations—that remain invisible under conventional, system-centric evaluation protocols. Our approach advances a paradigm shift from system-centered to user-centered IR evaluation, offering a fairness-aware, personalized assessment framework grounded in interpretable user modeling and supported by reproducible methodology.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Intelligent User Interfaces
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

This study proposes a method to diversify queries in existing test collections to reflect some of the diversity of search engine users, aligning with an earlier vision of an 'ideal' test collection. A Large Language Model (LLM) is used to create query variants: alternative queries that have the same meaning as the original. These variants represent user profiles characterised by different properties, such as language and domain proficiency, which are known in the IR literature to influence query formulation. The LLM's ability to generate query variants that align with user profiles is empirically validated, and the variants' utility is further explored for IR system evaluation. Results demonstrate that the variants impact how systems are ranked and show that user profiles experience significantly different levels of system effectiveness. This method enables an alternative perspective on system evaluation where we can observe both the impact of user profiles on system rankings and how system performance varies across users.

Problem

• Diversifying queries to reflect user diversity
• Using LLM to generate demographic query variants
• Evaluating IR systems across different user profiles

Innovation

• LLM generates demographic-based query variants
• Query variants represent diverse user proficiency profiles
• Method enables user-centric IR system evaluation