Analysis of AdvFusion: Adapter-based Multilingual Learning for Code Large Language Models

2025-11-03

AI Summary

This study investigates the effectiveness of AdvFusion—an adapter-based multilingual knowledge transfer method—in code large language models (Code-LLMs) across three cross-lingual tasks: code generation, code translation, and commit message generation. We systematically benchmark AdvFusion against prominent parameter-efficient fine-tuning (PEFT) methods, including AdapterFusion, LoRA, Compacter, and TaskAdapter. Results reveal that AdvFusion outperforms AdapterFusion in code generation but lags behind LoRA and Compacter; its advantage in code translation grows substantially with model scale; yet it underperforms in commit message generation. Crucially, this work is the first to empirically identify AdvFusion’s dual characteristics—task dependency and scale sensitivity—in multilingual Code-LLM adaptation. These findings provide both empirical evidence and methodological guidance for efficient, task-aware, and scalable cross-lingual adaptation of Code-LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Programming languages can benefit from one another by utilizing a language model for software engineering tasks. Full fine-tuning and Parameter Efficient Fine-Tuning (PEFT) of Code Language Models (Code-LMs) has been explored for multilingual knowledge transfer. AdapterFusion is a PEFT architecture that aims to enhance task performance by leveraging information from multiple programming languages, but primarily focuses on the target programming language. In our previous work, we proposed AdvFusion, a novel PEFT-based approach that effectively learns from other programming languages before adapting to the target task. Though previous experiments showed that AdvFusion outperformed AdapterFusion and LoRA, it was applied on pre-trained Code-LMs and was limited to only two tasks, code summarization and method name prediction. In this study, we expanded our work and investigated AdvFusion on Code Large Language Models (Code-LLMs), considering three new tasks: code generation, code translation, and commit message generation. We observed that different Code-LLMs/tasks exhibit different characteristics. In code generation, AdvFusion outperformed AdapterFusion but not other PEFT methods (LoRA, Compacter, and TaskAdapter). In commit message generation, AdapterFusion performed better than AdvFusion, and contrary to code generation, we found that the other PEFT methods do not have better performance. In code translation, AdvFusion performed worse than AdapterFusion overall, with the performance gap marginally widening as the model size increases. However, consistent with code generation, other PEFT methods showed better performance.

Problem

• Enhancing multilingual code transfer using adapter-based learning methods
• Evaluating AdvFusion performance across diverse software engineering tasks
• Comparing PEFT methods on code generation and translation tasks

Innovation

• Adapter-based multilingual learning for code models
• Transfer knowledge from multiple programming languages
• Parameter Efficient Fine-Tuning for Code-LLMs