Scalable Bayesian inference on high-dimensional multivariate linear regression

2025-08-22

AI Summary

Joint Bayesian estimation of the coefficient matrix and error precision matrix in high-dimensional multivariate linear regression remains challenging: existing approaches either violate positive definiteness (e.g., via generalized likelihoods) or yield only posterior modes—ignoring uncertainty—and suffer from poor scalability. Method: We propose a two-stage Bayesian inference framework, comprising an exact and an approximate variant. The exact method employs a spike-and-slab prior on coefficients and a DAG-Wishart prior on the precision matrix, ensuring positive definiteness and enabling joint posterior sampling. The approximate method decouples response dependencies to achieve scalable, consistent selection of both coefficients and the error DAG structure. Contribution/Results: We establish selection consistency for both the coefficient matrix and the error DAG, and derive posterior convergence rates. Empirical results demonstrate superior performance over state-of-the-art methods in prediction accuracy, graph structure recovery, and computational efficiency.

Technology Category

• Reasoning under Uncertainty: Graphical Models
• Machine Learning: Probabilistic Circuits and Graphical Models
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We consider jointly estimating the coefficient matrix and the error precision matrix in high-dimensional multivariate linear regression models. Bayesian methods in this context often face computational challenges, leading to previous approaches that either utilize a generalized likelihood without ensuring the positive definiteness of the precision matrix or rely on maximization algorithms targeting only the posterior mode, thus failing to address uncertainty. In this work, we propose two Bayesian methods: an exact method and an approximate two-step method. We first propose an exact method based on spike and slab priors for the coefficient matrix and DAG-Wishart prior for the error precision matrix, whose computational complexity is comparable to the state-of-the-art generalized likelihood-based Bayesian method. To further enhance scalability, a two-step approach is developed by ignoring the dependency structure among response variables. This method estimates the coefficient matrix first, followed by the calculation of the posterior of the error precision matrix based on the estimated errors. We validate the two-step method by demonstrating (i) selection consistency and posterior convergence rates for the coefficient matrix and (ii) selection consistency for the directed acyclic graph (DAG) of errors. We demonstrate the practical performance of proposed methods through synthetic and real data analysis.

Problem

• Scalable Bayesian inference for high-dimensional multivariate regression
• Estimating coefficient and precision matrices with computational efficiency
• Addressing uncertainty beyond posterior mode in Bayesian methods

Innovation

• Exact Bayesian method with spike-slab and DAG-Wishart priors
• Two-step approximate method for enhanced scalability
• Selection consistency and posterior convergence validation