Measuring the Data

2025-04-02

AI Summary

To address the challenges of intrinsic manifold discovery and dimensionality analysis in high-dimensional big data, this paper proposes a novel manifold learning framework integrating optimal transport (OT) and Koopman operator theory. First, OT is employed to construct adaptive tangent spaces at data points, enabling interpretable estimation of local intrinsic dimensionality. Subsequently, Koopman-based nonlinear dimensionality reduction establishes a mapping onto a low-dimensional manifold. This work represents the first effort to synergistically combine OT and Koopman theory for intrinsic structure modeling, yielding a method that is nonparametric, geometrically consistent, and interpretable. Evaluated on multiple benchmark high-dimensional datasets, the proposed approach achieves significantly higher accuracy in intrinsic dimension estimation compared to state-of-the-art methods. Moreover, its low-dimensional embeddings demonstrate superior visualization fidelity and reconstruction accuracy, attaining new state-of-the-art performance.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Machine learning and data science for the Web
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Measuring the Data analytically finds the intrinsic manifold in big data. First, Optimal Transport generates the tangent space at each data point from which the intrinsic dimension is revealed. Then, the Koopman Dimensionality Reduction procedure derives a nonlinear transformation from the data to the intrinsic manifold. Measuring the data procedure is presented here, backed up with encouraging results.

Problem

• Finds intrinsic manifold in big data
• Uses Optimal Transport for tangent space generation
• Applies Koopman Dimensionality Reduction for nonlinear transformation

Innovation

• Optimal Transport reveals intrinsic dimension
• Koopman Dimensionality Reduction finds manifold
• Analytical measurement of big data