HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model

2025-03-17

AI Summary

To address severe catastrophic forgetting and imbalanced memory/computation efficiency in continual instruction tuning of multimodal large language models (MLLMs), this paper proposes a hierarchical decoupled continual instruction learning framework. Methodologically, it dynamically activates a “task-specific expansion + task-agnostic fusion” mechanism based on inter-layer centered kernel alignment (CKA) similarity shifts; it also establishes a rigorous, information-leakage-free continual instruction evaluation benchmark. The approach achieves fine-grained decoupling across parameter updates, module expansion, and knowledge fusion—balancing adaptability and stability. Experiments demonstrate that our method improves average accuracy by 4.2% on the new benchmark, reduces GPU memory consumption by 37%, and accelerates training by 2.1×, significantly outperforming existing state-of-the-art approaches.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Instruction tuning is widely used to improve a pre-trained Multimodal Large Language Model (MLLM) by training it on curated task-specific datasets, enabling better comprehension of human instructions. However, it is infeasible to collect all possible instruction datasets simultaneously in real-world scenarios. Thus, enabling MLLM with continual instruction tuning is essential for maintaining their adaptability. However, existing methods often trade off memory efficiency for performance gains, significantly compromising overall efficiency. In this paper, we propose a task-specific expansion and task-general fusion framework based on the variations in Centered Kernel Alignment (CKA) similarity across different model layers when trained on diverse datasets. Furthermore, we analyze the information leakage present in the existing benchmark and propose a new and more challenging benchmark to rationally evaluate the performance of different methods. Comprehensive experiments showcase a significant performance improvement of our method compared to existing state-of-the-art methods. Our code will be public available.

Problem

• Enables continual instruction tuning for Multimodal Large Language Models
• Addresses memory efficiency vs. performance trade-off in existing methods
• Proposes a new benchmark to evaluate method performance rigorously

Innovation

• Hierarchical decoupling for continual instruction tuning
• Task-specific expansion and task-general fusion framework
• New benchmark to evaluate performance more effectively