Exact Schur-Sylvester Dimensionality Reductions for Non-Smooth Stochastic Complexity and Manifold Sampling

2026-06-22

AI Summary

This work addresses the 𝒪(N³) computational bottleneck in evaluating normalized maximum likelihood (NML) code lengths for nonsmooth estimators such as Lasso. The authors propose a geometric Propose-and-Project Metropolis–Hastings sampling framework that leverages block Schur complements, Sylvester’s determinant identity, and generalized KKT matrix inversion techniques to efficiently and accurately compute projection operators and volume factors. This approach reduces the NML code length complexity to 𝒪(k³ + N²k) for the first time and extends naturally to sparse SVM, elastic net, and group Lasso while preserving double-precision numerical equivalence. Empirical results demonstrate over 14,100× speedup on high-dimensional data, substantially enhancing the scalability and computational efficiency of NML estimation for nonsmooth regularized models.

Abstract

The exact computation of the Normalized Maximum Likelihood (NML) codelength for regular non-smooth estimators (e.g., Lasso) has been historically limited by the cubic scaling walls of manifold-constrained projection and volume integration. At each step of the geometric Propose-and-Project Metropolis--Hastings (PPMH) sampler, evaluating the projection operator requires inverting an $(N+k) \times (N+k)$ generalized KKT matrix, while calculating the volume factor requires the determinant of an $(N-k) \times (N-k)$ Gram matrix. This paper presents an exact, mathematically equivalent formulation that bypasses both bottlenecks by utilizing the block Schur complement and Sylvester's determinant identity. We prove that the computational complexity of both operations collapses from $\mathcal{O}(N^3)$ to $\mathcal{O}(k^3 + N^2 k)$ per step. We generalize this reduction to Sparse Support Vector Machines (SVMs), Elastic Net, and Group Lasso. Finally, we provide a rigorous numerical stability analysis and evaluate the sampler's efficiency using the Effective Sample Size (ESS) per second. Our empirical benchmarks on high-dimensional datasets confirm a constant speedup exceeding $14{,}100\times$ while maintaining double-precision numerical equivalence, rendering exact non-smooth NML estimation highly tractable for large-scale statistical inference.

Problem

• Normalized Maximum Likelihood
• Non-smooth Estimators
• Manifold Sampling
• Computational Complexity
• Dimensionality Reduction

Innovation

• Schur complement
• Sylvester's determinant identity
• Normalized Maximum Likelihood
• non-smooth estimators
• dimensionality reduction