Enhancing Multimodal Continual Instruction Tuning with BranchLoRA

2025-05-31

AI Summary

To address catastrophic forgetting in Mixture-of-Experts Low-Rank Adaptation (MoE-LoRA) for Multimodal Continual Instruction Tuning (MCIT), this paper proposes BranchLoRA—a branched LoRA architecture. Its key contributions are: (1) a heterogeneous asymmetric branching design that decouples task-specific and shared knowledge; (2) a tuning-free dynamic freezing mechanism to preserve branch specialization and mitigate cross-task interference; and (3) an incremental MoE task router coupled with an identifier-free task selector, enabling unbiased task routing and scalable extension. BranchLoRA maintains parameter efficiency while significantly improving cross-task knowledge retention. Experiments on the latest MCIT benchmark demonstrate that BranchLoRA consistently outperforms MoE-LoRA in both average performance and stability, and it is compatible with multimodal large language models of varying scales.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Multimodal Continual Instruction Tuning (MCIT) aims to finetune Multimodal Large Language Models (MLLMs) to continually align with human intent across sequential tasks. Existing approaches often rely on the Mixture-of-Experts (MoE) LoRA framework to preserve previous instruction alignments. However, these methods are prone to Catastrophic Forgetting (CF), as they aggregate all LoRA blocks via simple summation, which compromises performance over time. In this paper, we identify a critical parameter inefficiency in the MoELoRA framework within the MCIT context. Based on this insight, we propose BranchLoRA, an asymmetric framework to enhance both efficiency and performance. To mitigate CF, we introduce a flexible tuning-freezing mechanism within BranchLoRA, enabling branches to specialize in intra-task knowledge while fostering inter-task collaboration. Moreover, we incrementally incorporate task-specific routers to ensure an optimal branch distribution over time, rather than favoring the most recent task. To streamline inference, we introduce a task selector that automatically routes test inputs to the appropriate router without requiring task identity. Extensive experiments on the latest MCIT benchmark demonstrate that BranchLoRA significantly outperforms MoELoRA and maintains its superiority across various MLLM sizes.

Problem

• Mitigates Catastrophic Forgetting in continual multimodal instruction tuning
• Improves parameter efficiency in MoELoRA frameworks for MLLMs
• Enhances inter-task collaboration via flexible tuning-freezing mechanisms

Innovation

• BranchLoRA framework enhances efficiency and performance
• Flexible tuning-freezing mechanism mitigates catastrophic forgetting
• Task selector routes inputs without task identity