The Normalized Maximum Likelihood for Regular Non-Smooth Models: Measure-Theoretic Foundations and Geometric Sampling

2026-05-23

AI Summary

This work addresses the limitation of existing Normalized Maximum Likelihood (NML) methods, which are restricted to smooth models and cannot accommodate nonsmooth estimators such as Lasso. For the first time, it establishes a rigorous NML theoretical framework for the class of path-differentiable Lipschitz (PDL) nonsmooth models. By extending the coarea formula to nonsmooth settings and integrating tools from geometric measure theory with conservative Jacobian matrices, the approach enables exact computation of stochastic complexity. Furthermore, the paper introduces PDL-PPMH, a geometric MCMC algorithm that leverages automatic differentiation and a stochastic tangent-space proposal mechanism to efficiently traverse nonsmooth likelihood level sets. Experiments demonstrate that the method achieves exact posterior sampling for high-dimensional Lasso problems (p=2000), attaining prediction optimality comparable to cross-validation without requiring data splitting, thereby substantially improving data efficiency.

Abstract

The Normalized Maximum Likelihood (NML) codelength, or stochastic complexity, represents a principled criterion for universal coding. While recent coarea-based formulations provided a calculation method for smooth models, this framework collapses for the non-smooth estimators ubiquitous in modern machine learning (e.g., Lasso, Sparse SVMs). In this work, we provide a rigorous framework for computing the NML for regular path-differentiable Lipschitz (PDL) estimators. By applying classical geometric measure theory and bridging the coarea formula with conservative Jacobians, we prove that the stochastic complexity for non-smooth models is well-posed and theoretically consistent with the outputs of modern Automatic Differentiation. To compute this quantity exactly, we introduce the Propose-and-Project Metropolis-Hastings (PDL-PPMH) sampler, a geometric MCMC algorithm capable of traversing the non-differentiable level sets of the maximum likelihood estimator. We theoretically justify its components, including a stochastic tangent space proposal and a provably convergent non-smooth projection solver. We demonstrate the method's robustness by sampling from a high-dimensional Lasso posterior ($P=2000$), while simultaneously quantifying the computational scaling that governs the trade-off between exactness and mixing time. Crucially, we empirically demonstrate that our exact NML criterion provides a highly data-efficient alternative to cross-validation, achieving statistically indistinguishable predictive optima without requiring data splitting. Altogether, our work paves the way for the theoretical analysis of the NML codelength for regular non-smooth models.

Problem

• Normalized Maximum Likelihood
• Non-smooth Models
• Stochastic Complexity
• Geometric Measure Theory
• Automatic Differentiation

Innovation

• Normalized Maximum Likelihood
• Non-smooth Models
• Geometric Measure Theory
• Conservative Jacobians
• MCMC Sampling