BloomIntent: Automating Search Evaluation with LLM-Generated Fine-Grained User Intents

2025-09-23

AI Summary

Existing search evaluation methods struggle to capture users’ fine-grained and heterogeneous information intents, leading to inaccurate measurement of alignment between retrieval systems and actual user goals. To address this, we propose BloomIntent—the first intent-level evaluation framework that elevates assessment granularity from the query level to the intent level. BloomIntent generates interpretable, evaluable fine-grained intents by jointly modeling user attributes and information need types. It integrates a hierarchical intent taxonomy, semantic clustering, and large language model–driven automated scoring to construct a structured evaluation interface, enabling scalable, multi-dimensional intent analysis and insight generation. Experimental results show 72% agreement between BloomIntent-generated intents and expert judgments across three technical evaluations. A case study further demonstrates its effectiveness in uncovering unmet user needs, significantly improving intent alignment and practical utility in search evaluation.

Technology Category

• Search and Optimization: Evaluation and Analysis
• Data Mining & Knowledge Management: Intelligent Query Processing
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

If 100 people issue the same search query, they may have 100 different goals. While existing work on user-centric AI evaluation highlights the importance of aligning systems with fine-grained user intents, current search evaluation methods struggle to represent and assess this diversity. We introduce BloomIntent, a user-centric search evaluation method that uses user intents as the evaluation unit. BloomIntent first generates a set of plausible, fine-grained search intents grounded on taxonomies of user attributes and information-seeking intent types. Then, BloomIntent provides an automated evaluation of search results against each intent powered by large language models. To support practical analysis, BloomIntent clusters semantically similar intents and summarizes evaluation outcomes in a structured interface. With three technical evaluations, we showed that BloomIntent generated fine-grained, evaluable, and realistic intents and produced scalable assessments of intent-level satisfaction that achieved 72% agreement with expert evaluators. In a case study (N=4), we showed that BloomIntent supported search specialists in identifying intents for ambiguous queries, uncovering underserved user needs, and discovering actionable insights for improving search experiences. By shifting from query-level to intent-level evaluation, BloomIntent reimagines how search systems can be assessed -- not only for performance but for their ability to serve a multitude of user goals.

Problem

• Evaluating search systems for diverse user intents rather than query-level performance
• Automating assessment of search results against fine-grained user goals using LLMs
• Addressing limitations of current methods in representing user intent diversity

Innovation

• Generates fine-grained user intents from taxonomies
• Evaluates search results per intent using LLMs
• Clusters similar intents and summarizes outcomes