Sparse Covariance Neural Networks

2024-10-02

arXiv.org

AI Summary

To address performance degradation and computational inefficiency in Vector Neural Networks (VNNs) applied to tabular data graph convolution—caused by spurious correlations in the sample covariance matrix—this paper proposes a sparse preprocessing framework. Specifically, it employs hard-thresholding (for sparse ground-truth covariance) or soft-thresholding combined with random sparse sampling (for dense ground-truth covariance) to improve covariance estimation fidelity. We establish, for the first time, a theoretical connection between the stability of sparse-covariance VNNs and underlying data distribution properties. Furthermore, a dual-path sparsification mechanism is introduced to enhance model robustness. Experiments on brain functional connectivity analysis and human motion recognition demonstrate that our method achieves a 3.2% accuracy gain over baseline VNNs, accelerates training by 37%, and reduces cross-dataset performance variance by 58%.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Covariance Neural Networks (VNNs) perform graph convolutions on the covariance matrix of tabular data and achieve success in a variety of applications. However, the empirical covariance matrix on which the VNNs operate may contain many spurious correlations, making VNNs' performance inconsistent due to these noisy estimates and decreasing their computational efficiency. To tackle this issue, we put forth Sparse coVariance Neural Networks (S-VNNs), a framework that applies sparsification techniques on the sample covariance matrix before convolution. When the true covariance matrix is sparse, we propose hard and soft thresholding to improve covariance estimation and reduce computational cost. Instead, when the true covariance is dense, we propose stochastic sparsification where data correlations are dropped in probability according to principled strategies. We show that S-VNNs are more stable than nominal VNNs as well as sparse principal component analysis. By analyzing the impact of sparsification on their behavior, we provide novel connections between S-VNN stability and data distribution. We support our theoretical findings with experimental results on various application scenarios, ranging from brain data to human action recognition, and show an improved task performance, stability, and computational efficiency of S-VNNs compared with nominal VNNs.

Problem

• VNNs suffer from spurious correlations in empirical covariance matrices
• S-VNNs apply sparsification to improve covariance estimation and efficiency
• Framework addresses both sparse and dense true covariance scenarios

Innovation

• Sparse VNNs apply sparsification to covariance matrices
• Hard/soft thresholding used for sparse true covariance
• Stochastic sparsification employed for dense true covariance