The Effect of Document Selection on Query-focused Text Analysis

2026-04-13

AI Summary

This study addresses the lack of systematic evaluation regarding the impact of document selection strategies in query-focused text analysis, a gap that has led to ad hoc methodological choices. It establishes document selection as a critical methodological decision rather than merely a computational compromise and systematically evaluates seven selection strategies—ranging from random sampling to semantic and hybrid retrieval—across four prominent topic modeling approaches: LDA, BERTopic, TopicGPT, and HiCode. Experiments conducted on two datasets involving 26 open-ended queries demonstrate that semantic and hybrid retrieval strategies consistently achieve a robust balance between output quality and computational efficiency, warranting their recommendation as default choices for query-driven text analysis.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Search and Optimization: Evaluation and Analysis
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Web Mining and Content Analysis: Topic discovery and tracking

Abstract

Analyses of document collections often require selecting what data to analyze, as not all documents are relevant to a particular research question and computational constraints preclude analyzing all documents, yet little work has examined effects of selection strategy choices. We systematically evaluate seven selection methods (from random selection to hybrid retrieval) on outputs from four text analyses methods (LDA, BERTopic, TopicGPT, HiCode) over two datasets with 26 open-ended queries. Our evaluation reveals practice guidance: semantic or hybrid retrieval offer strong go-to approaches that avoid the pitfalls of weaker selection strategies and the unnecessary compute overhead of more complicated ones. Overall, our evaluation framework establishes data selection as a methodological decision, rather than a practical necessity, inviting the development of new strategies.

Problem

• document selection
• query-focused text analysis
• selection strategy
• text analysis methods
• data selection

Innovation

• document selection
• query-focused analysis
• topic modeling
• retrieval strategies
• evaluation framework