AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention

2026-08-12

AI Summary

This work addresses the issue of catastrophic forgetting in multimodal large language models during fine-tuning on downstream tasks, which often leads to significant degradation of pre-trained general capabilities. To mitigate this, the authors propose AWARe, a method that dynamically evaluates parameter importance based on activation signals and introduces an activation-weighted importance scoring mechanism. Without altering the model architecture, AWARe adaptively freezes critical parameters to preserve prior knowledge while updating the remaining parameters to accommodate new tasks. The approach is compatible with existing inference engines and demonstrates superior performance over current methods across multiple downstream tasks, effectively maintaining upstream capabilities while enhancing target task performance.

Abstract

Multimodal Large Language Models (MLLMs) exhibit strong generalization and reasoning abilities due to large-scale multimodal pre-training. However, fine-tuning these models on downstream tasks often leads to catastrophic forgetting, where newly learned task-specific knowledge degrades previously acquired capabilities. This issue arises because gradient updates for new tasks overwrite parameters critical to prior knowledge, limiting the practical deployment of MLLMs. To address this challenge, we propose Activation-Weighted Adaptive REtention (AWARe), a fine-tuning method that mitigates catastrophic forgetting by dynamically controlling parameter updates based on activation patterns. AWARe assigns activation-based importance scores to parameters, selectively freezing those essential for preserving prior capabilities while allowing less important parameters to adapt to new tasks. Importantly, AWARe operates without modifying model architectures, ensuring compatibility with existing inference engines. Extensive experiments demonstrate that AWARe effectively preserves upstream capabilities while achieving superior downstream performance compared to existing methods. Code is available at https://github.com/kaln27/AWARe.

Problem

• catastrophic forgetting
• Multimodal Large Language Models
• fine-tuning
• parameter overwriting
• prior knowledge preservation

Innovation

• catastrophic forgetting
• multimodal large language models
• activation-weighted retention
• parameter importance
• adaptive fine-tuning