A direct approach to tree-guided feature aggregation for high-dimensional regression

2025-07-25

AI Summary

To address the challenge of balancing interpretability, prediction accuracy, and rare-feature handling in high-dimensional linear models, this paper proposes a tree-guided equipartitioned feature aggregation method. The approach enforces structured sparsity via hierarchical overlapping group regularization and tree-structured constraints induced by a semi-norm—without requiring over-parameterization. We develop a non-iterative, one-step proximal algorithm compatible with diverse loss functions, including least squares and binomial deviance. A finite-sample error bound is established, with theoretical analysis showing convergence rates that are superior to or at least comparable with those of existing methods. Extensive simulations and microbiome data analyses demonstrate that the method substantially reduces estimation variability, improves feature aggregation fidelity, and enables valid post-selection statistical inference.

Technology Category

• Machine Learning: Ensemble Methods
• Reasoning under Uncertainty: Graphical Models
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

In high-dimensional linear models, sparsity is often exploited to reduce variability and achieve parsimony. Equi-sparsity, where one assumes that predictors can be aggregated into groups sharing the same effects, is an alternative parsimonious structure that can be more suitable in certain applications. Previous work has clearly demonstrated the benefits of exploiting equi-sparsity in the presence of ``rare features'' (Yan and Bien 2021). In this work, we propose a new tree-guided regularization scheme for simultaneous estimation and feature aggregation. Unlike existing methods, our estimator avoids synthetic overparameterization and its detrimental effects. Even though our penalty is applied to hierarchically overlapped groups, we show that its proximal operator can be solved with a one-pass, non-iterative algorithm. Novel techniques are developed to study the finite-sample error bound of this seminorm-induced regularizer under least squares and binomial deviance losses. Theoretically, compared to existing methods, the proposed method offers a faster or equivalent rate depending on the true equi-sparisty structure. Extensive simulation studies verify these findings. Finally, we illustrate the usefulness of the proposed method with an application to a microbiome dataset, where we conduct post-selection inference on the aggregated features' effects.

Problem

• Proposes tree-guided regularization for high-dimensional regression feature aggregation
• Avoids overparameterization with efficient one-pass proximal operator algorithm
• Provides theoretical error bounds and faster rates for equi-sparsity structures

Innovation

• Tree-guided regularization for feature aggregation
• Non-iterative proximal operator algorithm
• Finite-sample error bound analysis