Linear cost mutual information estimation and independence test of similar performance as HSIC

2025-08-25

AI Summary

Existing statistical dependence measures—such as the Hilbert–Schmidt Independence Criterion (HSIC)—exhibit prohibitive computational complexity (O(n².³⁷)) for large-scale data. To address this, we propose Hierarchical Correlation Reconstruction (HCR), a linear-time dependence measure. HCR constructs dependence representations using pairwise, ternary, and higher-order mixed-moment features; it estimates mutual information and performs independence testing in O(n) time via moment expansion approximations and efficient linear algorithms. Compared to HSIC, HCR achieves substantial computational speedup while improving sensitivity to subtle dependencies, enabling interpretable joint distribution modeling and multi-order dependence analysis. Empirical evaluations demonstrate that HCR matches or exceeds HSIC in dependence detection accuracy across diverse benchmarks. Thus, HCR establishes a new paradigm for dependence quantification in massive-data settings—offering efficiency, robustness, and interpretability.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Evaluation of statistical dependencies between two data samples is a basic problem of data science/machine learning, and HSIC (Hilbert-Schmidt Information Criterion)~cite{HSIC} is considered the state-of-art method. However, for size $n$ data sample it requires multiplication of $n imes n$ matrices, what currently needs $sim O(n^{2.37})$ computational complexity~cite{mult}, making it impractical for large data samples. We discuss HCR (Hierarchical Correlation Reconstruction) as its linear cost practical alternative of even higher dependence sensitivity in tests, and additionally providing actual joint distribution model by description of dependencies through features being mixed moments, starting with correlation and homoscedasticity, also allowing to approximate mutual information as just sum of squares of such nontrivial mixed moments between two data samples. Such single dependence describing feature is calculated in $O(n)$ linear time. Their number to test varies with dimension $d$ - requiring $O(d^2)$ for pairwise dependencies, $O(d^3)$ if wanting to also consider more subtle triplewise, and so on.

Problem

• Linear cost mutual information estimation for large datasets
• Independence testing with higher sensitivity than HSIC
• Hierarchical correlation reconstruction for dependency modeling

Innovation

• Linear cost mutual information estimation
• Hierarchical Correlation Reconstruction method
• Mixed moments features for dependencies