A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training

2026-08-26

AI Summary

本文提出视觉依赖感知框架，通过视觉约束最优传输和视觉调节适应方法解决多模态无监督持续后训练中的跨模态灾难性遗忘问题。

Abstract

In this paper, we explore a novel task of Multimodal Unsupervised Continual Post-Training (MU-CPT), enabling deployed MLLMs to continually evolve from streaming unlabeled data. Existing unsupervised post-training methods for MLLMs typically optimize target tokens uniformly, overlooking their heterogeneous visual dependence (VD). However, we reveal that token-level VD is crucial for MU-CPT. Specifically, its structural distortion serves as an indicator of cross-modal catastrophic forgetting, and its inherent heterogeneity acts as a compass to guide new-task learning. Leveraging this property, we propose a Visual Dependence-Aware (VDA) framework with two main components. First, Visually Constrained Optimal Transport (VC-OT) formulates the VD structural distortion of old-task VD during new-task learning as an optimal transport problem to mitigate cross-modal forgetting. By designing a region-aware ground cost and a dependence-stratified transport penalty, it prevents global shifts in visual focus while strictly prohibiting visual reliance from degenerating into language bias. Second, Visually Modulated Adaptation (VMA) exploits VD heterogeneity to emphasize visually grounded new-task learning, promoting new-task plasticity. Together, our method simultaneously maintains old-task stability and new-task plasticity during challenging MU-CPT. Extensive experiments under our MU-CPT setting validate the effectiveness of VDA.

Problem

• Multimodal Unsupervised Continual Post-Training
• Visual Dependence
• Cross-modal Catastrophic Forgetting

Innovation

• Visual Dependence-Aware
• Multimodal Unsupervised Continual Post-Training
• Visually Constrained Optimal Transport
• Visually Modulated Adaptation