Can Generative LLMs Create Query Variants for Test Collections? An Exploratory Study

2023-07-18

Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

AI Summary

This study investigates whether large language models (LLMs) can automatically generate high-quality search queries and their semantic variants from information need descriptions to enable low-cost, scalable test collection construction for information retrieval (IR). We propose a prompt-engineering-based LLM query generation method and design a multidimensional evaluation framework measuring semantic similarity, document pool coverage, and relevant document overlap. To our knowledge, this is the first systematic validation of LLM-generated variants for Chinese IR test collection construction. While LLM variants exhibit slightly lower diversity than human-annotated ones, they achieve a 71.1% relevant document overlap rate at pool depth 100—significantly outperforming baseline methods. Our results demonstrate that LLMs can effectively support automated, high-fidelity test collection creation, offering a novel, practical paradigm for IR evaluation infrastructure.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper explores the utility of a Large Language Model (LLM) to automatically generate queries and query variants from a description of an information need. Given a set of information needs described as backstories, we explore how similar the queries generated by the LLM are to those generated by humans. We quantify the similarity using different metrics and examine how the use of each set would contribute to document pooling when building test collections. Our results show potential in using LLMs to generate query variants. While they may not fully capture the wide variety of human-generated variants, they generate similar sets of relevant documents, reaching up to 71.1% overlap at a pool depth of 100.

Problem

• Large Language Models
• Search Term Generation
• Document Retrieval

Innovation

• Large Language Models
• Search Term Generation
• Information Retrieval