Cross Mutual Information

2025-07-21

AI Summary

Quantifying and comparing statistical dependence between random variables X and Y across distinct sample sets is challenging under non-stationary distributions, as conventional dependence measures—such as mutual information—lack comparability under distributional shift. Method: We propose Cross Mutual Information (CMI), the first formally defined, cross-sample comparable dependence measure. CMI jointly estimates the joint and marginal distributions from two independent samples to assess consistency in the X–Y dependence structure, thereby overcoming the fundamental incomparability of standard mutual information under distribution shift. Contribution/Results: We establish a tight theoretical connection between CMI and the coefficient of determination (R²) in linear regression. In extensive simulations across diverse nonlinear dependence structures, CMI consistently outperforms baseline methods and exhibits robustness to distribution drift. This framework provides an interpretable, generalizable tool for cross-subject or cross-task dependence modeling in high-dimensional non-stationary domains such as neuroimaging.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Multi-modal Vision
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Mutual information (MI) is a useful information-theoretic measure to quantify the statistical dependence between two random variables: $X$ and $Y$. Often, we are interested in understanding how the dependence between $X$ and $Y$ in one set of samples compares to another. Although the dependence between $X$ and $Y$ in each set of samples can be measured separately using MI, these estimates cannot be compared directly if they are based on samples from a non-stationary distribution. Here, we propose an alternative measure for characterising how the dependence between $X$ and $Y$ as defined by one set of samples is expressed in another, extit{cross mutual information}. We present a comprehensive set of simulation studies sampling data with $X$-$Y$ dependencies to explore this measure. Finally, we discuss how this relates to measures of model fit in linear regression, and some future applications in neuroimaging data analysis.

Problem

• Quantify dependence between X and Y in different sample sets
• Compare MI estimates from non-stationary distributions effectively
• Propose cross mutual information for cross-sample dependency analysis

Innovation

• Proposes cross mutual information measure
• Compares dependencies in non-stationary distributions
• Applies to neuroimaging data analysis