LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning

2025-08-08

AI Summary

To address the dual challenges of catastrophic forgetting and excessive parameter overhead in continual vision instruction tuning (CVIT) for multimodal large language models (MLLMs), this paper proposes an efficient architecture expansion method. Our approach introduces nested LoRA: it shares the LoRA matrix A across tasks, applies low-rank decomposition to matrix B, and incorporates cosine similarity regularization to stabilize training; additionally, task-specific module isolation ensures parameter-efficient and scalable continual learning. Experiments on multiple CVIT benchmarks demonstrate that our method reduces trainable parameters by 42% on average while consistently outperforming existing state-of-the-art methods. It achieves superior stability, generalization, and parameter efficiency—simultaneously mitigating forgetting and minimizing computational overhead.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Continual Visual Instruction Tuning (CVIT) enables Multimodal Large Language Models (MLLMs) to incrementally learn new tasks over time. However, this process is challenged by catastrophic forgetting, where performance on previously learned tasks deteriorates as the model adapts to new ones. A common approach to mitigate forgetting is architecture expansion, which introduces task-specific modules to prevent interference. Yet, existing methods often expand entire layers for each task, leading to significant parameter overhead and poor scalability. To overcome these issues, we introduce LoRA in LoRA (LiLoRA), a highly efficient architecture expansion method tailored for CVIT in MLLMs. LiLoRA shares the LoRA matrix A across tasks to reduce redundancy, applies an additional low-rank decomposition to matrix B to minimize task-specific parameters, and incorporates a cosine-regularized stability loss to preserve consistency in shared representations over time. Extensive experiments on a diverse CVIT benchmark show that LiLoRA consistently achieves superior performance in sequential task learning while significantly improving parameter efficiency compared to existing approaches.

Problem

• Mitigates catastrophic forgetting in continual visual instruction tuning
• Reduces parameter overhead in architecture expansion methods
• Improves scalability for multimodal large language models

Innovation

• Shares LoRA matrix A across tasks
• Applies low-rank decomposition to matrix B
• Uses cosine-regularized stability loss