Exploring Causes of Representational Similarity in Machine Learning Models

2025-05-20

AI Summary

This work investigates the causal origins of representation similarity in machine learning models, focusing on two key factors: dataset overlap and task overlap. Using large-scale, cross-modal, and multi-model experiments, we systematically disentangle and quantify their independent and synergistic effects on representation alignment. Representation similarity is measured via Centered Kernel Alignment (CKA), and linear regression-based attribution analysis is employed to isolate causal contributions. Results show that both dataset overlap and task overlap significantly increase representation similarity (p < 0.001), with their joint effect being strongest—providing the first causal empirical support for the Platonic Representation Hypothesis. To foster reproducibility and further research, we open-source all code and benchmark datasets. This work advances the understanding of representation interpretability and alignment mechanisms by establishing a rigorous, experimentally grounded causal framework.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Numerous works have noted significant similarities in how machine learning models represent the world, even across modalities. Although much effort has been devoted to uncovering properties and metrics on which these models align, surprisingly little work has explored causes of this similarity. To advance this line of inquiry, this work explores how two possible causal factors -- dataset overlap and task overlap -- influence downstream model similarity. The exploration of dataset overlap is motivated by the reality that large-scale generative AI models are often trained on overlapping datasets of scraped internet data, while the exploration of task overlap seeks to substantiate claims from a recent work, the Platonic Representation Hypothesis, that task similarity may drive model similarity. We evaluate the effects of both factors through a broad set of experiments. We find that both positively correlate with higher representational similarity and that combining them provides the strongest effect. Our code and dataset are published.

Problem

• Exploring causes of similarity in ML model representations
• Investigating impact of dataset and task overlap on model similarity
• Assessing correlation between overlap factors and representational similarity

Innovation

• Investigates dataset overlap impact on model similarity
• Examines task overlap influence on representation alignment
• Combines both factors for strongest similarity effect