Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation

2026-04-22

AI Summary

This work addresses the scarcity of multilingual benchmarks for Tip-of-the-Tongue (ToT) retrieval, which has been predominantly limited to English. We present the first large-scale multilingual ToT query simulation framework encompassing Chinese, Japanese, Korean, and English. Leveraging large language models and multilingual Wikipedia, we design language-aware prompting strategies to generate synthetic queries and systematically investigate how the language of prompts and source documents influences query fidelity. Validated through rank correlation with real user queries, our benchmark comprises 5,000 domain-diverse ToT queries per language. The study highlights the critical role of non-English corpora in query generation and establishes effective guidelines for constructing cross-lingual ToT datasets.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Tip-of-the-Tongue (ToT) retrieval benchmarks have largely focused on English, limiting their applicability to multilingual information access. In this work, we construct multilingual ToT test collections for Chinese, Japanese, Korean, and English, using an LLM-based query simulation framework. We systematically study how prompt language and source document language affect the fidelity of simulated ToT queries, validating synthetic queries through system rank correlation against real user queries. Our results show that effective ToT simulation requires language-aware design choices: non-English language sources are generally important, while English Wikipedia can be beneficial when non-English sources provide insufficient information for query generation. Based on these findings, we release four ToT test collections with 5,000 queries per language across multiple domains. This work provides the first large-scale multilingual ToT benchmark and offers practical guidance for constructing realistic ToT datasets beyond English.

Problem

• Tip-of-the-Tongue
• multilingual
• information retrieval
• query generation
• benchmark

Innovation

• multilingual retrieval
• tip-of-the-tongue query
• LLM-based simulation
• test collection construction
• cross-lingual information access