🤖 AI Summary
Existing tree-based models for tabular data generation lack explicit density modeling capabilities. Method: This paper proposes NRGBoost—the first energy-based generative gradient-boosting tree framework. It extends discriminative GBDT to a generative model capable of learning unnormalized densities (up to a constant) by optimizing an energy potential function via second-order Taylor expansion, where tree structure updates are driven jointly by gradients and Hessian matrices in a generative boosting scheme. Results: On multiple real-world tabular datasets, NRGBoost matches XGBoost’s discriminative performance while significantly outperforming other generative tree methods; its sampling quality rivals that of neural generative models, and it supports arbitrary conditional inference and joint sampling. The core contribution is the first realization of second-order optimization–driven generative tree boosting, unifying high discriminative accuracy with principled generative capability.
📝 Abstract
Despite the rise to dominance of deep learning in unstructured data domains, tree-based methods such as Random Forests (RF) and Gradient Boosted Decision Trees (GBDT) are still the workhorses for handling discriminative tasks on tabular data. We explore generative extensions of these popular algorithms with a focus on explicitly modeling the data density (up to a normalization constant), thus enabling other applications besides sampling. As our main contribution we propose an energy-based generative boosting algorithm that is analogous to the second-order boosting implemented in popular libraries like XGBoost. We show that, despite producing a generative model capable of handling inference tasks over any input variable, our proposed algorithm can achieve similar discriminative performance to GBDT on a number of real world tabular datasets, outperforming alternative generative approaches. At the same time, we show that it is also competitive with neural-network-based models for sampling. Code is available at https://github.com/ajoo/nrgboost.