Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning

2026-07-13

AI Summary

This work addresses catastrophic forgetting in federated multimodal large language models under dynamic data streams by proposing FedCMM, a novel framework that synergistically integrates modality-aware elastic weight consolidation, embedding-level data-free generative replay, and gradient similarity-aware aggregation. FedCMM preserves critical parameters through modality-separated Fisher information matrices, synthesizes multimodal replay samples via a lightweight local generative module, and adaptively weights client updates based on inter-task gradient cosine similarity. Without requiring access to raw client data, FedCMM significantly outperforms existing methods on two benchmarks, achieving state-of-the-art performance in both accuracy and backward transfer, thereby demonstrating robust continual learning capabilities in heterogeneous federated settings.

Abstract

Federated fine-tuning of Multimodal Large Language Models (MLLMs) across distributed networks enables privacy-sensitive adaptation to evolving data streams, yet a fundamental obstacle prevents robust deployment in dynamic environments: catastrophic forgetting, wherein sequential task updates erase previously acquired knowledge across visual, linguistic, and cross-modal representations. Addressing this challenge is especially critical for autonomous networked AI operating in safety-sensitive domains, such as content moderation, where reliable retention of prior knowledge underpins system integrity. To overcome this, we propose Federated Continual Multimodal Learning (FedCMM), a framework that embeds continual-learning safeguards into the federated optimization loop at three complementary levels. At the parameter level, modality-aware elastic weight consolidation computes separate Fisher information matrices for the vision encoder, language backbone, and cross-modal projector, providing granular, asymmetry-aware protection against modality-specific forgetting. At the data level, each client trains a lightweight local generative replay module to synthesize raw-data-free embedding-level multimodal replay tuples without any raw data sharing. At the aggregation level, Task-similarity-aware gradient aggregation autonomously filters and reweights client updates by gradient cosine similarity, suppressing conflicting directions and stabilizing the global learning trajectory. Extensive experiments on two benchmarks demonstrate that FedCMM consistently outperforms recent baselines on accuracy and backward transfer, confirming that holistic, modality-aware optimization enables robust evolutive adaptation across heterogeneous networked AI deployments.

Problem

• catastrophic forgetting
• federated fine-tuning
• multimodal large language models
• continual learning
• dynamic environments

Innovation

• Continual Learning
• Federated Learning
• Multimodal LLMs
• Elastic Regularization
• Synthetic Replay