MNAFT: modality neuron-aware fine-tuning of multimodal large language models for image translation

2026-04-18

AI Summary

This work addresses the challenges faced by multimodal large language models in image captioning tasks, where ineffective modeling of fine-grained vision-language alignment leads to a modality gap and full-parameter fine-tuning often causes catastrophic forgetting of pre-trained knowledge. To overcome these issues, the authors propose a modality-neuron-aware fine-tuning approach that leverages instruction-driven activation analysis to identify language-specific and language-agnostic neurons within visual and linguistic modules. Only critical neurons in task-relevant layers are selectively updated, while the rest of the parameters remain frozen to preserve pre-trained knowledge. This method pioneers the incorporation of neuronal functional specificity into multimodal adaptation, enabling precise parameter modulation based on modality and linguistic role. It achieves state-of-the-art performance across multiple image captioning benchmarks and offers visual insights into cross-modal understanding mechanisms.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal large language models (MLLMs) have shown impressive capabilities, yet they often struggle to effectively capture the fine-grained textual information within images crucial for accurate image translation. This often leads to a modality gap between visual text inputs and textual inputs/outputs for image translation. Existing methods, primarily relying on instruction fine-tuning, risk parameter redundancy of pre-trained knowledge, hindering generalization performance. To address this, we introduce modality neuron-aware fine-tuning (MNAFT), a novel approach that takes advantage of the specialized roles of individual neurons within MLLMs for enhanced image translation. MNAFT identifies language-agnostic and language-specific neurons in both vision and language modules through an instruction-driven activation analysis, evaluating their importance in various translation tasks. We then perform selective fine-tuning, updating only the parameters of language-specific and language-agnostic neurons within the selected layers relevant to the target task, while preserving the knowledge encoded in other neurons and layers. Our extensive experiments on multiple benchmarks demonstrate that MNAFT significantly outperforms state-of-the-art image translation methods, including cascaded models, standard full fine-tuning, and parameter-efficient tuning techniques. Furthermore, we provide comprehensive analysis, including visualizations of neuron activations and clustering patterns, to offer insights into the roles of different neuron groups in mediating cross-modal understanding and facilitating accurate language-specific translation.

Problem

• multimodal large language models
• image translation
• modality gap
• fine-grained textual information
• cross-modal understanding

Innovation

• modality neuron-aware fine-tuning
• multimodal large language models
• image translation
• selective fine-tuning
• cross-modal understanding