Rethinking Intrinsic Dimension Estimation in Neural Representations

2026-04-22

AI Summary

This study addresses a critical gap between theory and practice in estimating the intrinsic dimensionality (ID) of neural representations, revealing that prevailing ID estimators do not reliably capture the true ID but instead reflect local geometric properties of the representation space. Through rigorous theoretical analysis, numerical simulations, and empirical investigations, the work demonstrates that commonly reported ID values in the literature are primarily driven by such local structural characteristics rather than the underlying intrinsic dimensionality. Building on these insights, the paper reframes the interpretive framework for ID estimation, offering a more principled and nuanced perspective that advances the foundational understanding of dimensionality in neural representations.

Technology Category

• Machine Learning: Representation Learning
• Intelligent Robots: State Estimation
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

The analysis of neural representation has become an integral part of research aiming to better understand the inner workings of neural networks. While there are many different approaches to investigate neural representations, an important line of research has focused on doing so through the lens of intrinsic dimensions (IDs). Although this perspective has provided valuable insights and stimulated substantial follow-up research, important limitations of this approach have remained largely unaddressed. In this paper, we highlight a crucial discrepancy between theory and practice of IDs in neural representations, theoretically and empirically showing that common ID estimators are, in fact, not tracking the true underlying ID of the representation. We contrast this negative result with an investigation of the underlying factors that may drive commonly reported ID-related results on neural representation in the literature. Building on these insights, we offer a new perspective on ID estimation in neural representations.

Problem

• intrinsic dimension
• neural representations
• dimension estimation
• representation analysis

Innovation

• intrinsic dimension
• neural representations
• dimension estimation
• representation analysis
• estimation bias