Learning to Solve Related Linear Systems

2025-03-21

AI Summary

This work addresses the efficient solution of families of parameterized linear systems—such as symmetric positive definite systems arising repeatedly in hyperparameter optimization. We propose the first probabilistic linear solver framework operating in parameter space. The method models historical solution trajectories as a Gaussian process prior and employs Bayesian regression to infer, for a new parameter value, both the posterior mean (serving as a high-quality initial guess for preconditioned conjugate gradient—PCG) and the posterior covariance (used to construct an adaptive low-rank preconditioner). Compared to standard PCG, our approach reduces iteration counts by 30–50% across multiple numerical benchmarks. It is the first method to enable knowledge transfer and joint optimization across parameterized systems while preserving numerical stability and significantly improving computational efficiency.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Solving multiple parametrised related systems is an essential component of many numerical tasks. Borrowing strength from the solved systems and learning will make this process faster. In this work, we propose a novel probabilistic linear solver over the parameter space. This leverages information from the solved linear systems in a regression setting to provide an efficient posterior mean and covariance. We advocate using this as companion regression model for the preconditioned conjugate gradient method, and discuss the favourable properties of the posterior mean and covariance as the initial guess and preconditioner. We also provide several design choices for this companion solver. Numerical experiments showcase the benefits of using our novel solver in a hyperparameter optimisation problem.

Problem

• Solving multiple parametrised related linear systems efficiently
• Leveraging solved systems for faster learning and computation
• Improving preconditioned conjugate gradient method with probabilistic models

Innovation

• Probabilistic linear solver over parameter space
• Companion regression model for conjugate gradient
• Posterior mean and covariance as preconditioner