Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models

2026-05-14

AI Summary

This work addresses the challenge of catastrophic forgetting in multimodal large language models under continual learning scenarios, where existing approaches often rely on storing historical data, incur high computational costs, or employ insufficient regularization. To overcome these limitations, the authors propose Octopus, a novel framework that introduces, for the first time, a history-free gradient orthogonalization mechanism. Octopus decouples task adaptation from regularization through a two-stage fine-tuning process, constraining the direction of parameter updates to minimize interference between tasks—without retaining past data or altering the model architecture. Evaluated on the UCIT benchmark, Octopus achieves state-of-the-art performance, surpassing the previous best method by 2.14% and 6.82% on the Avg and Last metrics, respectively, thereby significantly improving the balance between model stability and plasticity.

Abstract

Continual learning in multimodal large language models (MLLMs) aims to sequentially acquire knowledge while mitigating catastrophic forgetting, yet existing methods face inherent limitations: architecture-based approaches incur additional computational overhead and often generalize poorly to new tasks, rehearsal-based methods rely on storing historical data, raising privacy and storage concerns, and conventional regularization-based strategies alone are insufficient to fully prevent parameter interference. We propose Octopus, a two-stage continual learning framework based on History-Free Gradient Orthogonalization (HiFGO), which enforces gradient-level orthogonality without historical task data. Our proposed two-stage finetuning strategy decouples task adaptation from regularization, achieving a principled balance between plasticity and stability. Experiments on UCIT show that Octopus establishes state-of-the-art performance, surpassing prior SOTA by 2.14% and 6.82% in terms of Avg and Last.

Problem

• continual learning
• multimodal large language models
• catastrophic forgetting
• parameter interference
• gradient orthogonalization

Innovation

• Gradient Orthogonalization
• Continual Learning
• Multimodal Large Language Models
• Catastrophic Forgetting
• History-Free Regularization