🤖 AI Summary
This study addresses the challenge that XGBoost struggles to effectively capture ratio-based interaction signals through feature splits in the presence of strong confounding factors. The authors systematically investigate the impact of intra-tree column subsampling (colsample_bylevel/bynode) on this capability, proposing a novel “shared-path” metric to quantify how well the model learns ratio structures that require coordinated splits. Through controlled experiments on synthetic data, evaluated using PR-AUC, they demonstrate that when only raw features are used, column subsampling can degrade test performance by up to 54% in PR-AUC. However, this performance loss is largely mitigated when explicit ratio features are incorporated. The work reveals the detrimental effect of column subsampling on modeling ratio interactions and provides both an interpretable quantitative tool and a practical mitigation strategy.
📝 Abstract
Many applied problems contain signal that becomes clear only after combining multiple raw measurements. Ratios and rates are common examples. In gradient boosted trees, this combination is not an explicit operation: the model must synthesize it through coordinated splits on the component features. We study whether intra-tree column subsampling in XGBoost makes that synthesis harder. We use two synthetic data generating processes with cancellation-style structure. In both, two primitive features share a strong nuisance factor, while the target depends on a smaller differential factor. A log ratio cancels the nuisance and isolates the signal. We vary colsample_bylevel and colsample_bynode over s in {0.4, 0.6, 0.8, 0.9}, emphasizing mild subsampling (s>= 0.8). A control feature set includes the engineered ratio, removing the need for synthesis. Across both processes, intra-tree column subsampling reduces test PR-AUC in the primitives-only setting. In the main process the relative decrease reaches 54 percent when both parameters are set to 0.4. The effect largely disappears when the engineered ratio is present. A path-based co-usage metric drops in the same cells where performance deteriorates. Practically, if ratio-like structure is plausible, either avoid intra-tree subsampling or include the intended ratio features.