Optimizing Multi-Modal Trackers via Sensitivity-aware Regularized Tuning

2025-08-24

AI Summary

To address the plasticity–stability trade-off in fine-tuning RGB-pretrained models for multimodal tracking, this paper proposes a sensitivity-aware regularization framework. We introduce, for the first time, tangent-space modeling of parameter prior sensitivity and jointly incorporate transfer sensitivity during cross-modal adaptation, establishing a dual-sensitivity regularization mechanism that preserves original knowledge stability while enhancing cross-modal adaptability. Our method integrates tangent-space sensitivity quantification, dynamic regularization, and cross-modal transfer learning to achieve balanced generalization and plasticity. Extensive experiments demonstrate state-of-the-art performance across multiple multimodal tracking benchmarks. The source code and pretrained models are publicly released.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper tackles the critical challenge of optimizing multi-modal trackers by effectively adapting the pre-trained models for RGB data. Existing fine-tuning paradigms oscillate between excessive freedom and over-restriction, both leading to a suboptimal plasticity-stability trade-off. To mitigate this dilemma, we propose a novel sensitivity-aware regularized tuning framework, which delicately refines the learning process by incorporating intrinsic parameter sensitivities. Through a comprehensive investigation from pre-trained to multi-modal contexts, we identify that parameters sensitive to pivotal foundational patterns and cross-domain shifts are primary drivers of this issue. Specifically, we first analyze the tangent space of pre-trained weights to measure and orient prior sensitivities, dedicated to preserving generalization. Then, we further explore transfer sensitivities during the tuning phase, emphasizing adaptability and stability. By incorporating these sensitivities as regularization terms, our method significantly enhances the transferability across modalities. Extensive experiments showcase the superior performance of the proposed method, surpassing current state-of-the-art techniques across various multi-modal tracking. The source code and models will be publicly available at https://github.com/zhiwen-xdu/SRTrack.

Problem

• Optimizing multi-modal trackers by adapting pre-trained RGB models
• Addressing suboptimal plasticity-stability trade-off in fine-tuning
• Enhancing cross-modal transferability through sensitivity-aware regularization

Innovation

• Sensitivity-aware regularized tuning framework
• Incorporates intrinsic parameter sensitivities
• Enhances transferability across modalities