Covariance Scattering Transforms

2025-11-12

AI Summary

Existing covariance-driven methods (e.g., PCA) struggle to capture low-variance directional information and suffer from instability under small-sample regimes or when eigenvalues are clustered. While covariance neural networks (VNNs) offer improved robustness and expressivity, they require supervised training. To address these limitations, we propose the Covariance Scattering Transform (CST), an unsupervised, training-free deep representation learning framework. CST pioneers the integration of covariance wavelets into spectral-domain filtering, constructing stable, discriminative hierarchical representations in the covariance spectrum via layered local spectral filtering, nonlinear spectral transformations, and feature pruning. CST inherits PCA’s simplicity while achieving VNN-level robustness—particularly against covariance estimation errors in small-sample settings. Evaluated on age prediction using cortical thickness data from four neurodegenerative disease cohorts, CST matches or surpasses complex supervised models, demonstrating both efficacy and strong generalization capability.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Social Networks and Social Media: Computational social science

Abstract

Machine learning and data processing techniques relying on covariance information are widespread as they identify meaningful patterns in unsupervised and unlabeled settings. As a prominent example, Principal Component Analysis (PCA) projects data points onto the eigenvectors of their covariance matrix, capturing the directions of maximum variance. This mapping, however, falls short in two directions: it fails to capture information in low-variance directions, relevant when, e.g., the data contains high-variance noise; and it provides unstable results in low-sample regimes, especially when covariance eigenvalues are close. CoVariance Neural Networks (VNNs), i.e., graph neural networks using the covariance matrix as a graph, show improved stability to estimation errors and learn more expressive functions in the covariance spectrum than PCA, but require training and operate in a labeled setup. To get the benefits of both worlds, we propose Covariance Scattering Transforms (CSTs), deep untrained networks that sequentially apply filters localized in the covariance spectrum to the input data and produce expressive hierarchical representations via nonlinearities. We define the filters as covariance wavelets that capture specific and detailed covariance spectral patterns. We improve CSTs'computational and memory efficiency via a pruning mechanism, and we prove that their error due to finite-sample covariance estimations is less sensitive to close covariance eigenvalues compared to PCA, improving their stability. Our experiments on age prediction from cortical thickness measurements on 4 datasets collecting patients with neurodegenerative diseases show that CSTs produce stable representations in low-data settings, as VNNs but without any training, and lead to comparable or better predictions w.r.t. more complex learning models.

Problem

• Captures meaningful patterns in low-variance directions and noisy data
• Improves stability in low-sample regimes with close covariance eigenvalues
• Generates expressive hierarchical representations without requiring labeled training data

Innovation

• Deep untrained networks using localized covariance filters
• Covariance wavelets capturing detailed spectral patterns
• Pruning mechanism improving computational and memory efficiency