Scalable Computations for Generalized Mixed Effects Models with Crossed Random Effects Using Krylov Subspace Methods

2025-05-14

AI Summary

To address the computational bottleneck induced by Cholesky decomposition in high-dimensional crossed random-effects generalized linear mixed models (GLMMs), this paper proposes a novel Krylov-subspace-based algorithmic framework. We systematically design an adaptive preconditioner tailored for conjugate gradient methods and stochastic Lanczos quadrature, rigorously derive its convergence bounds, and thereby enable efficient and numerically stable estimation of prediction variances. The method integrates Krylov iterations, stochastic quadratic-form estimation, and a high-performance C++ implementation with Python/R interfaces. Experiments demonstrate approximately 100× speedup over standard Cholesky-based solvers; on both synthetic and real-world datasets, our implementation—GPBoost—outperforms lme4 and glmmTMB by up to four orders of magnitude while markedly improving numerical stability. The core contribution is overcoming scalability limitations posed by high-cardinality categorical variables, establishing the first iterative GLMM solver that simultaneously offers theoretical guarantees and practical performance for large-scale mixed-effects modeling.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Mixed effects models are widely used for modeling data with hierarchically grouped structures and high-cardinality categorical predictor variables. However, for high-dimensional crossed random effects, current standard computations relying on Cholesky decompositions can become prohibitively slow. In this work, we present novel Krylov subspace-based methods that address several existing computational bottlenecks. Among other things, we theoretically analyze and empirically evaluate various preconditioners for the conjugate gradient and stochastic Lanczos quadrature methods, derive new convergence results, and develop computationally efficient methods for calculating predictive variances. Extensive experiments using simulated and real-world data sets show that our proposed methods scale much better than Cholesky-based computations, for instance, achieving a runtime reduction of approximately two orders of magnitudes for both estimation and prediction. Moreover, our software implementation is up to 10'000 times faster and more stable than state-of-the-art implementations such as lme4 and glmmTMB when using default settings. Our methods are implemented in the free C++ software library GPBoost with high-level Python and R packages.

Problem

• Address computational bottlenecks in mixed effects models
• Improve scalability for high-dimensional crossed random effects
• Enhance speed and stability compared to existing implementations

Innovation

• Uses Krylov subspace methods for scalability
• Develops efficient preconditioners for convergence
• Implements fast C++ library with Python/R