Meta-Learning Preferences for Multilingual LLM Alignment

2026-07-14

AI Summary

This work addresses the challenge of aligning multilingual large language models with human preferences in low-resource languages, where preference data is scarce. To overcome this limitation, the paper introduces a cross-lingual meta-learning framework that integrates reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO), leveraging high-resource languages to learn transferable initial policies. The proposed approach significantly enhances alignment efficiency for low-resource languages. Theoretical analysis provides convergence guarantees, and empirical results demonstrate that with only 100 target-language preference samples, the method achieves up to a 28% improvement in win rate across diverse languages and model scales, consistently outperforming existing baselines.

Abstract

Unequal availability of human preference data across languages poses a significant challenge for aligning large language models in multilingual settings. To address the lack of sufficient data in low-resource language alignment, we propose a meta-learning framework for Reinforcement Learning from Human Feedback and Direct Preference Optimization. By leveraging preference data from other languages, our framework learns a transferable initialization that enables effective adaptation to a target language with minimal data. We provide theoretical guarantees for both the meta-reward modeling and meta-policy optimization settings, and empirically demonstrate the effectiveness of our approach on multilingual benchmarks. In an extremely low-resource setting with only 100 target-language preference samples, our approach achieves up to $28\%$ win-rate improvements over baseline methods, and consistently outperforms baselines across multiple target languages and model scales. Our approaches retain these advantages across different combinations of meta-training languages and varying linguistic distances from the target languages.

Problem

• multilingual alignment
• preference data scarcity
• low-resource languages
• human feedback
• language model alignment

Innovation

• meta-learning
• multilingual alignment
• preference optimization
• low-resource adaptation
• reinforcement learning from human feedback