TRACER: Persistent Regularization for Robust Multimodal Finetuning

2026-05-28

AI Summary

This work addresses the degradation of out-of-distribution (OOD) robustness in fine-tuned pretrained multimodal models, often caused by catastrophic forgetting. To mitigate this issue, the authors propose a multi-view self-distillation approach that integrates contrastive learning with a weighted moving average (WMA)-guided teacher mechanism, establishing a theoretical framework for multimodal contrastive fine-tuning. The study reveals that standard exponential moving average (EMA) teachers are prone to representation collapse during robust fine-tuning, and accordingly introduces the WMA teacher to provide continual regularization, enabling unbiased convergence within the task subspace while preserving orthogonal knowledge. Evaluated across three backbone architectures, the method consistently yields significant improvements in OOD accuracy and calibration performance, demonstrating strong robustness to hyperparameter choices.

Abstract

Mainstream strategies for finetuning pretrained multimodal models often degrade out-of-distribution (OOD) robustness, a phenomenon known as catastrophic forgetting. In this paper, we develop a theoretical framework for multimodal contrastive finetuning, yielding closed-form solutions and a geometric decomposition for each strategy. This framework shows that self-distillation is more effective than other regularization approaches to retain the knowledge of the pretrained model. Our analysis reveals a largely overlooked limitation: standard Exponential Moving Average (EMA) teachers, widely used in robust finetuning, suffer from collapse. To solve this, we prove that a Weighted Moving Average (WMA) teacher maintains a persistent regularizing force over finite horizons and yields bias-free convergence in the task subspace while preserving orthogonal knowledge. These insights motivate **TRACER** (**T**rajectory-**R**obust **A**nchoring for **C**ontrastive **E**ncoder **R**egularization), which combines contrastive learning with WMA-guided multi-perspective distillation. Extensive experiments on CLIP finetuning demonstrate consistent OOD accuracy and calibration gains across three backbone architectures, and comprehensive ablations confirm that TRACER is both principled and robust to hyperparameter choices. Code is available at [https://github.com/HesamAsad/TRACER](https://github.com/HesamAsad/TRACER).

Problem

• catastrophic forgetting
• out-of-distribution robustness
• multimodal finetuning
• teacher collapse
• pretrained knowledge retention

Innovation

• contrastive finetuning
• catastrophic forgetting
• weighted moving average
• multimodal regularization
• out-of-distribution robustness