Curvature-Guided Mixing for MLLM Adaptation

2026-06-23

AI Summary

This work addresses the catastrophic forgetting of general capabilities in multimodal large language models after task-specific fine-tuning. To mitigate this issue, the authors propose the Curvature-Guided Merging (CGM) framework, which, for the first time, incorporates second-order curvature information—approximated via the Hessian of the loss landscape—into model merging. CGM analytically derives a task-adaptive soft parameter merging ratio and further introduces CGM†, a sparse hard-merging variant based on curvature-aware scoring. The method provides a theoretically grounded, closed-form optimal merging strategy. Experiments on LLaVA-1.5 and Qwen2.5-VL demonstrate that CGM significantly outperforms existing approaches, effectively preserving general-purpose abilities while enhancing downstream task performance.

Abstract

Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities. Existing model merging methods to combat this are often heuristic or use sub-optimal objectives. We propose CurvatureGuided Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models. CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form "soft mixing" ratio. This ratio intelligently blends parameters based on their relative task-specific curvatures. We also introduce CGM$\dagger$, a robust "hard mixing" variant that performs sparse parameter selection guided by a novel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen2.5VL across multiple downstream tasks show that CGM and CGM$\dagger$ consistently improve the trade-off between task specialization and general knowledge retention over existing methods. Code is available at github.com/zzsyjl/CGM-ECCV-2026.

Problem

• catastrophic forgetting
• Multimodal Large Language Models
• model merging
• fine-tuning
• general capabilities

Innovation

• Curvature-Guided Mixing
• Model Merging
• Catastrophic Forgetting
• Hessian Approximation
• Multimodal Large Language Models