xgboost

Existing tree-based models for tabular data generation lack explicit density modeling capabilities. Method: This paper proposes NRGBoost—the first energy-based generative gradient-boosting tree framework. It extends discriminative GBDT to a generative model capable of learning unnormalized densities (up to a constant) by optimizing an energy potential function via second-order Taylor expansion, where tree structure updates are driven jointly by gradients and Hessian matrices in a generative boosting scheme. Results: On multiple real-world tabular datasets, NRGBoost matches XGBoost’s discriminative performance while significantly outperforming other generative tree methods; its sampling quality rivals that of neural generative models, and it supports arbitrary conditional inference and joint sampling. The core contribution is the first realization of second-order optimization–driven generative tree boosting, unifying high discriminative accuracy with principled generative capability.

This paper systematically evaluates six linear programming (LP)-based fully-corrective boosting methods—including two newly proposed approaches, NM-Boost and QRLP-Boost—against mainstream heuristic gradient boosting frameworks (e.g., XGBoost, LightGBM) across 20 benchmark datasets. Addressing limitations in accuracy, ensemble sparsity, margin distribution, real-time inference efficiency, and hyperparameter robustness, the study provides the first large-scale empirical evidence that LP-based boosters achieve accuracy competitive with state-of-the-art gradient boosting when using shallow decision trees as base learners, while drastically reducing ensemble size (sparsity improved by multiple-fold). Crucially, these methods enable lossless sparsification of pre-trained ensembles. The results demonstrate that global optimization—central to LP-based boosting—enhances both robustness and interpretability, revealing a promising new paradigm for trustworthy ensemble learning.

Gradient-boosted trees (GBTs) struggle to adapt to the non-stationary data distributions inherent in online reinforcement learning (RL). Method: This paper proposes GBRL—a novel framework that introduces GBTs into online RL for the first time, supporting actor-critic architectures and policy optimization. It innovatively incorporates tree-structured parameter sharing, per-parameter adaptive learning rates, and GPU acceleration—preserving GBTs’ native compatibility with structured and categorical features while enhancing interpretability and deployment efficiency. Contribution/Results: GBRL achieves performance on par with deep neural networks across diverse RL benchmarks, yet yields significantly more compact models. We release an open-source, high-performance GBRL library (GitHub: NVlabs/gbrl), fully compatible with mainstream RL frameworks such as Stable-Baselines3. GBRL bridges a critical gap in integrating gradient boosting with online RL, enabling scalable, interpretable, and efficient tree-based RL.

Traditional gradient boosting trees (GBTs) employ static tree structures and fixed splitting criteria, limiting their adaptability to evolving gradient distributions and task-specific阶段性 characteristics during training. To address this, we propose MorphBoost—a novel GBT framework featuring self-organizing, dynamic tree structures. Its core contributions are: (1) a gradient-statistics-driven adaptive splitting function that adjusts split decisions based on evolving gradient moments; (2) problem-fingerprint identification guided by training progress, enabling dynamic regulation of learning pressure across stages; and (3) vectorized tree prediction coupled with interaction-aware feature importance modeling. Evaluated on 10 standard benchmark datasets, MorphBoost achieves an average accuracy gain of 0.84% over XGBoost and attains state-of-the-art performance on 40% of the datasets. Moreover, it significantly reduces prediction variance and improves minimum accuracy, demonstrating superior stability and generalization capability.

To bridge the theoretical gap between decision trees and boosting methods—and address their high generalization error—this paper proposes Guided Random Forests (GRAF). GRAF is the first random forest variant to incorporate a global partitioning mechanism, synergizing oblique decision trees to achieve both locally optimal splits and globally optimized tree structures. It introduces a “local splitting → global ensemble” strategy, unifying support for classification and data approximation tasks. Methodologically, GRAF reconciles the bias–variance trade-off discrepancy between single-tree models and boosting algorithms, while establishing the first random-forest-based paradigm for data approximation. Extensive experiments across 115 benchmark datasets demonstrate that GRAF achieves statistically significant improvements—or competitive performance—over state-of-the-art ensemble methods (e.g., XGBoost, LightGBM, standard RF) in both classification accuracy and generalization capability. Crucially, GRAF’s theoretical generalization error bound is strictly lower than prior forest models, with empirical validation confirming this reduction.

This work proposes a gradient boosting framework tailored for vector-valued outputs, addressing the limitations of conventional approaches that rely on dimension-wise updates or diagonal Hessian approximations and thus fail to capture interdependencies among output dimensions. By incorporating histogram-accelerated decision trees capable of supporting non-diagonal Hessian approximations and employing vector-valued leaf nodes, the proposed method enables joint modeling of the output structure. This approach relaxes the simplifying assumptions commonly imposed on vector targets in existing algorithms, substantially enhancing model expressiveness and predictive accuracy while preserving computational efficiency during training.

This study addresses the critical dependence of tree boosting models’ generalization performance on hyperparameter configuration, a domain lacking systematic comparison and guiding principles among existing tuning methods. Conducting a large-scale empirical evaluation across 59 regression and classification datasets under a unified experimental framework, the authors compare prominent hyperparameter optimization approaches—including random search, grid search, Tree-structured Parzen Estimator (TPE), Gaussian process Bayesian optimization, Hyperband, and SMAC. The findings reveal that SMAC consistently outperforms other methods across most tasks, while default hyperparameters yield suboptimal results. Notably, all hyperparameters significantly influence performance, contradicting the common assumption that only a few are critical. Effective tuning requires extensive sampling (>100 trials), and for regression tasks, employing early stopping proves superior to including the number of iterations within the search space.

This work addresses the challenge of efficiently generating mixed-type tabular data by proposing XGenBoost, the first framework to effectively adapt XGBoost for generative modeling. The approach comprises two models: an XGBoost-driven denoising diffusion implicit model (DDIM) tailored for small datasets and a hierarchical autoregressive model designed for large-scale data. A key innovation is a Gaussian–multinomial joint diffusion mechanism that operates without one-hot encoding, combined with empirical quantile function-based dequantization and hierarchical classifiers to preserve the ordinal structure of numerical features. Evaluated across multiple benchmarks, XGenBoost consistently outperforms existing neural and tree-based generative models in generation quality while substantially reducing training costs.

This work proposes the first agnostic boosting algorithm that simultaneously achieves near-optimal sample complexity and polynomial runtime. In the assumption-free agnostic setting, boosting has long faced a fundamental trade-off between statistical efficiency and computational tractability. Addressing this challenge, the authors introduce a novel polynomial-time algorithm grounded in agnostic learning theory. When all other parameters are fixed, the algorithm’s runtime scales polynomially with the sample size—a significant improvement over prior methods that required exponential time. This advancement marks the first unification of computational efficiency and near-optimal sample complexity in agnostic boosting, substantially enhancing its practical applicability while maintaining theoretical guarantees.

This study addresses the challenge that XGBoost struggles to effectively capture ratio-based interaction signals through feature splits in the presence of strong confounding factors. The authors systematically investigate the impact of intra-tree column subsampling (colsample_bylevel/bynode) on this capability, proposing a novel “shared-path” metric to quantify how well the model learns ratio structures that require coordinated splits. Through controlled experiments on synthetic data, evaluated using PR-AUC, they demonstrate that when only raw features are used, column subsampling can degrade test performance by up to 54% in PR-AUC. However, this performance loss is largely mitigated when explicit ratio features are incorporated. The work reveals the detrimental effect of column subsampling on modeling ratio interactions and provides both an interpretable quantitative tool and a practical mitigation strategy.