MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models

2024-02-21

arXiv.org

AI Summary

Large language models (LLMs) frequently generate hallucinated content, and cross-lingual reliability assessment remains unexplored. Method: We introduce MlingConf, a multilingual reliability evaluation benchmark comprising language-agnostic (LA) and language-specific (LS) tasks. It features a high-quality, manually annotated multilingual dataset and establishes— for the first time—a unified framework for multilingual confidence estimation. Contribution/Results: Our analysis reveals a pronounced language-dominance effect in English for LA tasks. For LS tasks, native-tone prompting—i.e., eliciting confidence estimates using native-language instructions—improves average accuracy by 12.3%. This simple, efficient strategy significantly enhances LLMs’ culturally adaptive reliability, offering both a novel methodological approach and empirical foundation for trustworthy multilingual AI.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The tendency of Large Language Models (LLMs) to generate hallucinations raises concerns regarding their reliability. Therefore, confidence estimations indicating the extent of trustworthiness of the generations become essential. However, current LLM confidence estimations in languages other than English remain underexplored. This paper addresses this gap by introducing a comprehensive investigation of Multilingual Confidence estimation (MlingConf) on LLMs, focusing on both language-agnostic (LA) and language-specific (LS) tasks to explore the performance and language dominance effects of multilingual confidence estimations on different tasks. The benchmark comprises four meticulously checked and human-evaluated high-quality multilingual datasets for LA tasks and one for the LS task tailored to specific social, cultural, and geographical contexts of a language. Our experiments reveal that on LA tasks English exhibits notable linguistic dominance in confidence estimations than other languages, while on LS tasks, using question-related language to prompt LLMs demonstrates better linguistic dominance in multilingual confidence estimations. The phenomena inspire a simple yet effective native-tone prompting strategy by employing language-specific prompts for LS tasks, effectively improving LLMs' reliability and accuracy in LS scenarios.

Problem

• Investigates multilingual confidence estimation gaps in non-English LLMs
• Explores language dominance effects in confidence estimation tasks
• Proposes native-tone prompting to improve LS task reliability

Innovation

• Multilingual confidence estimation benchmark MlingConf
• Native-tone prompting for language-specific tasks
• Language-agnostic vs language-specific performance analysis