Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs

2026-06-22

AI Summary

This work addresses the challenge of catastrophic forgetting in multimodal large language models during continual learning, which often arises from drift in cross-modal attention patterns. The authors propose the first replay-free continual learning framework that treats cross-attention maps as two-dimensional signals and extracts their spectral statistical features to construct lightweight, skill-level prototype distributions. A phase-invariant spectral regularization term is introduced to constrain harmful attention drift without requiring storage of original data or generation of pseudo-samples. By innovatively leveraging the spectral properties of cross-modal attention, the method achieves significant performance gains over existing replay-based, regularization-based, and adapter-based approaches across multiple benchmarks—including VQA v2, VQACL, CLT-VQA, CoIN, and UCIT—effectively mitigating forgetting while enhancing final task performance.

Abstract

Multimodal large language models (MLLMs) are increasingly required to adapt to non-stationary streams of visual domains, question types, and user instructions, yet continual fine-tuning often causes severe forgetting of previously acquired multimodal skills. Existing continual vision-language methods mainly preserve outputs, replay data or pseudo-data, regularize embedding geometry, or allocate task-specific parameters, but they provide limited control over how internal cross-modal attention patterns supporting old skills drift during adaptation. We propose Attention-Spectrum Regularization (ASR), a replay-free continual learning framework that preserves skill-conditioned structures of cross-modal attention. ASR treats cross-attention maps as two-dimensional signals, summarizes their scale and directional properties into compact spectral statistics, and stores only skill-wise prototype distributions instead of replaying past image-question pairs, generated pseudo-examples, or old-stage teacher snapshots. In later stages, a phase-invariant spectral regularizer constrains harmful drift of these prototypes while allowing instance-level attention to adapt to new tasks. We provide theoretical analysis showing that skill-conditioned spectral drift controls forgetting under a spectral sufficiency assumption, and that Fourier power spectra are stable to spatial translations and bounded perturbations. Experiments on continual VQA and multimodal instruction-tuning benchmarks, including VQA v2, VQACL, CLT-VQA, CoIN, and UCIT, show that ASR consistently improves final performance and reduces forgetting over strong replay-, regularization-, and adapter-based baselines. Preserving skill-level attention structure is an effective and lightweight mechanism for continual MLLMs. Code is available at https://github.com/Creative-zcx/attention-spectrum-replay

Problem

• continual learning
• multimodal LLMs
• catastrophic forgetting
• cross-modal attention
• attention drift

Innovation

• Attention-Spectrum Regularization
• Continual Learning
• Multimodal LLMs
• Cross-Modal Attention
• Replay-Free