Revealing the Parallel Multilingual Learning within Large Language Models

2024-03-14

Conference on Empirical Methods in Natural Language Processing

AI Summary

It remains unclear whether parallel multilingual input (PMI)—simultaneous presentation of source text and its translations—enhances the understanding and reasoning capabilities of multilingual large language models (MLLMs). Method: We conduct a comprehensive empirical investigation using multilingual translation ensembling, neuron activation analysis, contrastive in-context learning evaluation, and interpretable neural probing. Contribution/Results: We provide the first empirical evidence that PMI significantly improves MLLM performance across diverse multilingual benchmarks, outperforming monolingual in-context learning. Mechanistically, PMI enables synergistic integration of source and translated inputs, inducing a ~23% reduction in neuronal activation entropy at critical layers—yielding more focused, robust, and compact representations. This compression parallels synaptic pruning in human neurodevelopment. Our findings uncover an implicit multilingual co-learning mechanism in MLLMs under PMI, establishing a novel, interpretable paradigm for efficient multilingual understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) can handle multilingual and cross-lingual text within a single input; however, previous works leveraging multilingualism in LLMs primarily focus on using English as the pivot language to enhance language understanding and reasoning. Given that multiple languages are a compensation for the losses caused by a single language’s limitations, it’s a natural next step to enrich the model’s learning context through the integration of the original input with its multiple translations. In this paper, we start by revealing that LLMs learn from parallel multilingual input (PMI). Our comprehensive evaluation shows that PMI enhances the model’s comprehension of the input, achieving superior performance than conventional in-context learning (ICL). Furthermore, to explore how multilingual processing affects prediction, we examine the activated neurons in LLMs. Surprisingly, involving more languages in the input activates fewer neurons, leading to more focused and effective neural activation patterns. Also, this neural reaction coincidently mirrors the neuroscience insight about synaptic pruning, highlighting a similarity between artificial and biological ‘brains’.

Problem

• Enhancing multilingual LLM comprehension via parallel multi-language input
• Testing PiM impact on 8 datasets, 7 languages, and 8 LLMs
• Exploring neuron activation patterns in PiM versus monolingual input

Innovation

• Parallel Input in Multiple Languages (PiM) enhances comprehension
• PiM inhibits neurons for precise activation
• Multilingual translation improves model performance significantly