Understanding Cross-Modal Contributions in Continual Vision-Language Models: A Theoretical Perspective

2026-06-12

AI Summary

This work addresses the challenge that existing continual learning methods for vision-language models struggle to balance adaptation to new tasks with retention of previously acquired knowledge, and lack a theoretical understanding of cross-modal contribution mechanisms. The paper proposes the first theoretical framework to systematically analyze how visual and linguistic modalities contribute during continual learning, revealing how task order and similarity influence modality robustness. Through extensive empirical evaluation on large-scale vision-language models, the study quantifies cross-modal contributions under diverse conditions and validates the efficacy of the proposed framework. Results demonstrate that the approach significantly enhances both the generalization capability and the robustness of modality-specific contributions in continual learning settings.

Abstract

Continual vision-language models are commonly addressed through sequential fine-tuning; however, although this paradigm enables adaptation to new environments (tasks), it inherently emphasizes the contribution of previously learned environments (tasks) at the expense of the stability required to preserve previously acquired knowledge. While existing approaches have adequately studied continual learning and catastrophic forgetting in vision-language models (VLMs), the theoretical understanding of modality-specific contributions across a sequence of environments remains largely unexplored. In this paper, we present a new theoretical perspective to understand the cross-modal (vision-language) contributions to consecutive environments. We empirically evaluate our theoretical findings on large VLMs and demonstrate their effectiveness in capturing environment-level cross-modal contributions. Our analysis provides deeper insights into continual VLMs, highlighting their contribution robustness to varying task orders and inter-task similarities, and their improved generalization performance.

Problem

• continual learning
• vision-language models
• cross-modal contributions
• catastrophic forgetting
• modality-specific contributions

Innovation

• cross-modal contributions
• continual vision-language models
• theoretical analysis
• catastrophic forgetting
• modality-specific robustness