PAC-Bayes Bounds for Multivariate Linear Regression and Linear Autoencoders

2025-12-14

AI Summary

Linear autoencoders (LAEs) achieve empirical success in recommender systems, yet lack rigorous generalization guarantees. Method: This paper establishes the first convergent PAC-Bayes generalization bound for multivariate linear regression and bridges LAEs to rank-constrained multivariate regression via exact theoretical equivalence. It further develops an efficient constrained optimization algorithm to scale the bound to large-scale real-world scenarios. Contribution/Results: The derived bound is tight and strongly correlated with ranking metrics—including Recall@K and NDCG@K—demonstrating both theoretical soundness and practical utility across multiple large-scale recommendation benchmarks. Key contributions are: (1) the first convergent PAC-Bayes bound for multivariate linear regression; (2) a rigorous theoretical equivalence between LAEs and rank-constrained regression; and (3) a generalization analysis framework that balances theoretical rigor with computational tractability.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Linear Autoencoders (LAEs) have shown strong performance in state-of-the-art recommender systems. However, this success remains largely empirical, with limited theoretical understanding. In this paper, we investigate the generalizability -- a theoretical measure of model performance in statistical learning -- of multivariate linear regression and LAEs. We first propose a PAC-Bayes bound for multivariate linear regression, extending the earlier bound for single-output linear regression by Shalaeva et al., and establish sufficient conditions for its convergence. We then show that LAEs, when evaluated under a relaxed mean squared error, can be interpreted as constrained multivariate linear regression models on bounded data, to which our bound adapts. Furthermore, we develop theoretical methods to improve the computational efficiency of optimizing the LAE bound, enabling its practical evaluation on large models and real-world datasets. Experimental results demonstrate that our bound is tight and correlates well with practical ranking metrics such as Recall@K and NDCG@K.

Problem

• Extends PAC-Bayes bounds to multivariate linear regression for theoretical generalization analysis.
• Applies these bounds to linear autoencoders by interpreting them as constrained regression models.
• Develops efficient methods to compute bounds for large-scale models and real-world datasets.

Innovation

• PAC-Bayes bound for multivariate linear regression
• Interpret LAEs as constrained regression on bounded data
• Optimize bound computationally for large-scale evaluation