Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression

2025-02-14

AI Summary

In deep heteroscedastic regression, the true covariance matrix is typically unknown and thus unavailable for direct supervision. To address this challenge, this paper proposes the first self-supervised learning framework that requires no covariance labels. Our method jointly parameterizes the predictive covariance via a neural network and optimizes it using a novel self-supervised loss. Key contributions include: (1) deriving the first differentiable and optimizable upper bound on the 2-Wasserstein distance for non-commutative Gaussian distributions—yielding a theoretically consistent unsupervised objective for covariance estimation; and (2) introducing a lightweight, neighborhood-similarity-driven pseudo-labeling mechanism to mitigate the absence of explicit covariance supervision. Evaluated on multiple synthetic and real-world benchmarks, our approach achieves prediction accuracy comparable to fully supervised methods while significantly reducing computational overhead.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Deep heteroscedastic regression models the mean and covariance of the target distribution through neural networks. The challenge arises from heteroscedasticity, which implies that the covariance is sample dependent and is often unknown. Consequently, recent methods learn the covariance through unsupervised frameworks, which unfortunately yield a trade-off between computational complexity and accuracy. While this trade-off could be alleviated through supervision, obtaining labels for the covariance is non-trivial. Here, we study self-supervised covariance estimation in deep heteroscedastic regression. We address two questions: (1) How should we supervise the covariance assuming ground truth is available? (2) How can we obtain pseudo labels in the absence of the ground-truth? We address (1) by analysing two popular measures: the KL Divergence and the 2-Wasserstein distance. Subsequently, we derive an upper bound on the 2-Wasserstein distance between normal distributions with non-commutative covariances that is stable to optimize. We address (2) through a simple neighborhood based heuristic algorithm which results in surprisingly effective pseudo labels for the covariance. Our experiments over a wide range of synthetic and real datasets demonstrate that the proposed 2-Wasserstein bound coupled with pseudo label annotations results in a computationally cheaper yet accurate deep heteroscedastic regression.

Problem

• Self-supervised covariance estimation in regression
• Addressing heteroscedasticity in deep learning models
• Developing efficient pseudo label generation algorithms

Innovation

• Self-supervised covariance estimation method
• 2-Wasserstein distance upper bound derived
• Neighborhood heuristic for pseudo labels