On the Degrees of Freedom of some Lasso procedures

2025-11-26

AI Summary

This paper addresses the lack of theoretical characterizations for the effective degrees of freedom (EDF) of adaptive Lasso and adaptive group Lasso. We propose, for the first time, unbiased EDF estimators under both orthogonal and non-orthogonal designs. Building upon the Stein’s unbiased risk estimation framework and leveraging structural analysis of penalized regression, we rigorously derive explicit closed-form EDF expressions, revealing the coupled influence of regularization parameters, coefficient signs, and least-squares initial estimates on model complexity. The proposed estimators require neither resampling nor approximation, offering both analytical tractability and broad applicability. Empirical evaluation on synthetic and real-world datasets demonstrates that our EDF estimator significantly improves model selection consistency of information criteria (e.g., AIC, BIC) and enhances the accuracy of prediction error estimation. This work provides a critical theoretical and practical tool for assessing and deploying adaptive regularization methods.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Graphical Models
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The effective degrees of freedom of penalized regression models quantify the actual amount of information used to generate predictions, playing a pivotal role in model evaluation and selection. Although a closed-form estimator is available for the Lasso penalty, adaptive extensions of widely used penalized approaches, including the Adaptive Lasso and Adaptive Group Lasso, have remained without analogous theoretical characterization. This paper presents the first unbiased estimator of the effective degrees of freedom for these methods, along with their main theoretical properties, for both orthogonal and non-orthogonal designs, derived within Stein's unbiased risk estimation framework. The resulting expressions feature inflation terms influenced by the regularization parameter, coefficient signs, and least-squares estimates. These advances enable more accurate model selection criteria and unbiased prediction error estimates, illustrated through synthetic and real data. These contributions offer a rigorous theoretical foundation for understanding model complexity in adaptive regression, bridging a critical gap between theory and practice.

Problem

• Estimates effective degrees of freedom for Adaptive Lasso and Group Lasso.
• Provides unbiased estimator for model complexity in adaptive regression.
• Enables accurate model selection and prediction error estimation.

Innovation

• Unbiased estimator for adaptive Lasso degrees of freedom
• Derived using Stein's unbiased risk estimation framework
• Enables accurate model selection and prediction error estimates