Quantum-inspired Benchmark for Estimating Intrinsic Dimension

2025-10-01

AI Summary

Existing intrinsic dimension estimation (IDE) methods exhibit substantial disagreement on complex manifolds, and no benchmark exists to systematically evaluate their performance under topologically nontrivial structures, variable curvature, and noise. This work introduces QuIIEst—the first quantum-optics-inspired benchmark for IDE evaluation, grounded in homogeneous space embeddings. QuIIEst generates an infinite family of topologically nontrivial manifolds with controllable non-uniform curvature, fractal geometry (e.g., the Hofstadter butterfly), and realistic noise models. It provides an interpretable, reproducible framework for rigorous IDE assessment. Experiments reveal that state-of-the-art IDE methods suffer significant accuracy degradation on QuIIEst and exhibit poor robustness to curvature variation; only a few methods correctly recover the effective dimension of fractal structures. QuIIEst thus establishes a critical evaluation standard for both theoretical analysis and practical deployment of IDE techniques.

Technology Category

• Machine Learning: Learning with Manifolds
• Intelligent Robots: State Estimation
• Search and Optimization: Non-convex Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Machine learning models can generalize well on real-world datasets. According to the manifold hypothesis, this is possible because datasets lie on a latent manifold with small intrinsic dimension (ID). There exist many methods for ID estimation (IDE), but their estimates vary substantially. This warrants benchmarking IDE methods on manifolds that are more complex than those in existing benchmarks. We propose a Quantum-Inspired Intrinsic-dimension Estimation (QuIIEst) benchmark consisting of infinite families of topologically non-trivial manifolds with known ID. Our benchmark stems from a quantum-optical method of embedding arbitrary homogeneous spaces while allowing for curvature modification and additive noise. The IDE methods tested were generally less accurate on QuIIEst manifolds than on existing benchmarks under identical resource allocation. We also observe minimal performance degradation with increasingly non-uniform curvature, underscoring the benchmark's inherent difficulty. As a result of independent interest, we perform IDE on the fractal Hofstadter's butterfly and identify which methods are capable of extracting the effective dimension of a space that is not a manifold.

Problem

• Estimating intrinsic dimension on complex manifolds
• Benchmarking accuracy of dimension estimation methods
• Testing methods on non-manifold fractal structures

Innovation

• Quantum-inspired benchmark for intrinsic dimension estimation
• Topologically non-trivial manifolds with known dimensions
• Curvature modification and additive noise integration