Scalable branch-and-bound model selection with non-monotonic criteria including AIC, BIC and Mallows's $mathit{C_p}$

2025-12-13

AI Summary

In high-dimensional model selection, non-monotonicity of information criteria (e.g., AICc, BIC, Cp) renders exhaustive search computationally intractable, while heuristic methods (e.g., stepwise regression) lack global optimality guarantees. This paper introduces the first tight, computationally tractable pruning bounds for non-monotonic information criteria—overcoming the long-standing belief that non-monotonicity precludes branch-and-bound optimization—and proposes a rigorous branch-and-bound algorithm that guarantees globally optimal model selection. Our approach integrates information-theoretic analysis with combinatorial optimization to tightly control computational complexity. Evaluated on a real-world task involving 4 billion candidate models, the method achieves over 6,000× speedup versus exhaustive search while provably delivering the globally optimal solution. This advance significantly enhances scalability and reliability for large-scale scientific modeling.

Technology Category

• Search and Optimization: Non-convex Optimization
• Constraint Satisfaction and Optimization: Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Model selection is a pivotal process in the quantitative sciences, where researchers must navigate between numerous candidate models of varying complexity. Traditional information criteria, such as the corrected Akaike Information Criterion (AICc), Bayesian Information Criterion (BIC), and Mallows's $mathit{C_p}$, are valuable tools for identifying optimal models. However, the exponential increase in candidate models with each additional model parameter renders the evaluation of these criteria for all models -- a strategy known as exhaustive, or brute-force, searches -- computationally prohibitive. Consequently, heuristic approaches like stepwise regression are commonly employed, albeit without guarantees of finding the globally-optimal model. In this study, we challenge the prevailing notion that non-monotonicity in information criteria precludes bounds on the search space. We introduce a simple but novel bound that enables the development of branch-and-bound algorithms tailored for these non-monotonic functions. We demonstrate that our approach guarantees identification of the optimal model(s) across diverse model classes, sizes, and applications, often with orders of magnitude computational speedups. For instance, in one previously-published model selection task involving $2^{32}$ (approximately 4 billion) candidate models, our method achieves a computational speedup exceeding 6,000. These findings have broad implications for the scalability and effectiveness of model selection in complex scientific domains.

Problem

• Develops branch-and-bound algorithm for non-monotonic model selection criteria
• Addresses computational infeasibility of exhaustive searches over many candidate models
• Ensures globally-optimal model identification with significant speedup over brute-force

Innovation

• Branch-and-bound algorithm for non-monotonic criteria
• Novel bound enabling scalable model selection
• Guarantees optimal model identification with speedups