The Triangle of Similarity: A Multi-Faceted Framework for Comparing Neural Network Representations

2026-01-23

AI Summary

This work proposes the “similarity triangle” framework to address the limitations of existing neural representation comparison methods, which often adopt a narrow perspective and fail to comprehensively capture the similarity of internal model mechanisms. The framework unifies three complementary dimensions—static (e.g., CKA or Procrustes), functional (e.g., linear probing connectivity and prediction similarity), and sparsity-based (pruning-induced robustness)—to systematically evaluate the representational geometry of CNNs, Vision Transformers (ViTs), and vision-language models. Experiments reveal that model architecture predominantly governs representational similarity, yielding distinct architectural family clusters on ImageNetV2 and CIFAR-10. Furthermore, CKA self-similarity strongly correlates with task performance, and pruning not only uncovers shared computational cores across models but also exerts a representational regularization effect.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Symbolic Representations

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Comparing neural network representations is essential for understanding and validating models in scientific applications. Existing methods, however, often provide a limited view. We propose the Triangle of Similarity, a framework that combines three complementary perspectives: static representational similarity (CKA/Procrustes), functional similarity (Linear Mode Connectivity or Predictive Similarity), and sparsity similarity (robustness under pruning). Analyzing a range of CNNs, Vision Transformers, and Vision-Language Models using both in-distribution (ImageNetV2) and out-of-distribution (CIFAR-10) testbeds, our initial findings suggest that: (1) architectural family is a primary determinant of representational similarity, forming distinct clusters; (2) CKA self-similarity and task accuracy are strongly correlated during pruning, though accuracy often degrades more sharply; and (3) for some model pairs, pruning appears to regularize representations, exposing a shared computational core. This framework offers a more holistic approach for assessing whether models have converged on similar internal mechanisms, providing a useful tool for model selection and analysis in scientific research.

Problem

• neural network representations
• representational similarity
• model comparison
• internal mechanisms
• scientific validation

Innovation

• Triangle of Similarity
• Representational Similarity
• Linear Mode Connectivity
• Model Pruning
• Neural Network Comparison