Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation

2026-04-21

AI Summary

This study investigates whether function vectors in multilingual large language models exhibit language independence and generalization capabilities in machine translation. Leveraging a decoder-only architecture, the authors extract function vectors from monolingual data and evaluate their efficacy through ablation studies and cross-lingual transfer experiments. The work provides the first empirical validation that function vectors can be effectively transferred across different languages and model variants—including instruction-tuned models—and demonstrates partial generalization from word-level to sentence-level translation tasks. Experimental results show that these vectors significantly improve the ranking of correct translation tokens for unseen languages; their removal degrades translation performance without affecting unrelated tasks, thereby confirming both their task specificity and cross-lingual validity.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Function vectors (FVs) are vector representations of tasks extracted from model activations during in-context learning. While prior work has shown that multilingual model representations can be language-agnostic, it remains unclear whether the same holds for function vectors. We study whether FVs exhibit language-agnosticity, using machine translation as a case study. Across three decoder-only multilingual LLMs, we find that translation FVs extracted from a single English$\rightarrow$Target direction transfer to other target languages, consistently improving the rank of correct translation tokens across multiple unseen languages. Ablation results show that removing the FV degrades translation across languages with limited impact on unrelated tasks. We further show that base-model FVs transfer to instruction-tuned variants and partially generalize from word-level to sentence-level translation.

Problem

• function vectors
• language-agnosticity
• machine translation
• multilingual LLMs
• in-context learning

Innovation

• function vectors
• language-agnosticity
• machine translation
• in-context learning
• cross-lingual transfer