Selected Languages are All You Need for Cross-lingual Truthfulness Transfer

2024-06-20

International Conference on Computational Linguistics

AI Summary

This work addresses the significant decline in factuality of multilingual large language models (LLMs) on non-English languages. We propose Fact-aware Multilingual Selective Synergy (FaMSS), the first framework to extend factuality evaluation to multilingual settings. FaMSS models language-specific biases and quantifies per-language contribution to adaptively select an optimal language subset. It then enables cross-lingual transfer of factuality knowledge via translation instruction tuning and multilingual alignment. Experiments across 12 languages show FaMSS improves average factuality scores by 18.7%, substantially reduces multilingual representation discrepancies, and notably enhances performance on low-resource languages. Our core contributions are: (i) a language-aware selective synergy mechanism that dynamically weights languages based on their factual reliability, and (ii) a factuality-driven multilingual instruction tuning paradigm that jointly optimizes translation fidelity and factual consistency across languages.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Truthfulness stands out as an essential challenge for Large Language Models (LLMs). Although many works have developed various ways for truthfulness enhancement, they seldom focus on truthfulness in multilingual scenarios. Meanwhile, contemporary multilingual aligning technologies struggle to balance numerous languages and often exhibit serious truthfulness gaps across different languages, especially those that differ greatly from English. In our work, we extend truthfulness evaluation to multilingual contexts and propose a practical method for cross-lingual truthfulness transfer called Fact-aware Multilingual Selective Synergy (FaMSS). FaMSS is able to select an optimal subset of all tested languages by language bias and transfer contributions, and then employ translation instruction tuning for cross-lingual truthfulness transfer. Experimental results demonstrate that our approach can effectively reduce the multilingual representation disparity and boost cross-lingual truthfulness transfer of LLMs.

Problem

• Addresses truthfulness gaps in multilingual LLMs.
• Proposes FaMSS for cross-lingual truthfulness transfer.
• Reduces multilingual representation disparity effectively.

Innovation

• Extends truthfulness evaluation to multilingual contexts
• Proposes Fact-aware Multilingual Selective Synergy (FaMSS)
• Employs translation instruction tuning for truthfulness transfer