Variance-Aware Loss Scheduling for Multimodal Alignment in Low-Data Settings

2025-03-05

AI Summary

In few-shot vision-language alignment, contrastive learning suffers from overfitting and training instability. To address this, we propose a dynamic loss scheduling method based on predictive variance in the embedding space: for the first time, cross-modal embedding prediction variance is adopted as an uncertainty metric to adaptively weight contrastive losses. This strategy outperforms conventional entropy- or cosine-similarity-based scheduling mechanisms, significantly enhancing alignment robustness and discriminability under data-scarce conditions. Experiments on a Flickr8k subset demonstrate substantial improvements in image–text retrieval accuracy; t-SNE visualizations reveal tighter cross-modal clustering; and noise robustness evaluations confirm slower recall degradation. Our work establishes an interpretable, scalable, uncertainty-driven optimization paradigm for few-shot vision-language modeling.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Multimodal Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Training vision-language models for image-text alignment typically requires large datasets to achieve robust performance. In low-data scenarios, standard contrastive learning can struggle to align modalities effectively due to overfitting and unstable training dynamics. In this paper, we propose a variance-aware loss scheduling approach that dynamically adjusts the weighting of the contrastive loss based on the statistical variability (uncertainty) in the model's alignment predictions. Using a subset of the Flickr8k image-caption dataset to simulate limited data conditions, we demonstrate that our approach improves image-text retrieval accuracy compared to a fixed-weight baseline. We also compare against other adaptive weighting strategies (using output entropy and cosine similarity spread) and find that variance-aware scheduling provides the best overall trade-off. Qualitatively, our method yields more distinct multimodal embeddings as shown by t-SNE visualizations. Moreover, in a stress test with noise-injected captions and images, the variance-guided loss proves more robust, maintaining higher recall when random perturbations are introduced. These results highlight the benefit of adaptive loss weighting for multimodal alignment in low-data regimes.

Problem

• Improves image-text alignment in low-data settings
• Dynamically adjusts contrastive loss based on prediction variability
• Enhances robustness against noise in multimodal embeddings

Innovation

• Variance-aware loss scheduling for alignment
• Dynamic contrastive loss weighting adjustment
• Improved robustness in low-data scenarios