Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion

2026-04-23

AI Summary

This study investigates why certain visual stimuli elicit more consistent cross-modal representations between vision and language models than others. To address this, the authors propose quantifying intra-modal dispersion of internal visual representations at the single-stimulus level and employ generalized Procrustes analysis to assess its impact on cross-modal alignment. They report the first evidence that stimuli with lower intra-modal dispersion achieve substantially higher cross-modal alignment—up to twofold improvement—a finding that holds robustly across diverse vision-language model pairs (e.g., DINOv2) and under varying stimulus selection criteria. These results highlight the critical role of intra-modal consistency in facilitating cross-modal convergence, offering a novel perspective on the mechanisms underlying representational alignment in both artificial and biological neural systems.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Neural networks exhibit a remarkable degree of representational convergence across diverse architectures, training objectives, and even data modalities. This convergence is predictive of alignment with brain representation. A recent hypothesis suggests this arises from learning the underlying structure in the environment in similar ways. However, it is unclear how individual stimuli elicit convergent representations across networks. An image can be perceived in multiple ways and expressed differently using words. Here, we introduce a methodology based on the Generalized Procrustes Algorithm to measure intra-modal representational convergence at the single-stimulus level. We applied this to vision models with distinct training objectives, selecting stimuli based on their degree of alignment (intra-modal dispersion). Crucially, we found that this intra-modal dispersion strongly modulates alignment between vision and language models (cross-modal convergence). Specifically, stimuli with low intra-modal dispersion (high agreement among vision models) elicited significantly higher cross-modal alignment than those with high dispersion, by up to a factor of two (e.g., in pairings of DINOv2 with language models). This effect was robust to stimulus selection criteria and generalized across different pairings of vision and language models. Measuring convergence at the single-stimulus level provides a path toward understanding the sources of convergence and divergence across modalities, and between neural networks and human neural representations.

Problem

• cross-modal convergence
• intra-modal dispersion
• single-stimulus representation
• neural alignment
• representational convergence

Innovation

• cross-modal convergence
• intra-modal dispersion
• single-stimulus analysis
• Generalized Procrustes Algorithm
• vision-language alignment