🤖 AI Summary
This work addresses the underperformance of large language models (LLMs) compared to gradient-boosted trees like XGBoost in few-shot tabular classification tasks. It introduces, for the first time, a boosting-inspired paradigm into LLM fine-tuning by proposing a parameter-efficient training framework. The method iteratively trains lightweight adapters as weak learners through multiple rounds of residual optimization and incorporates structured inductive bias by fusing decision tree paths with original tabular features to form a dual-view input representation. This enables an adaptive transition from path-guided to feature-driven representations. Experimental results demonstrate that the proposed approach significantly outperforms standard fine-tuning across diverse LLMs and tabular datasets, matches or even surpasses XGBoost under varying sample sizes, and achieves superior performance to GPT-4o using a 4B-parameter model.
📝 Abstract
Large language models (LLMs) have recently been adapted to tabular prediction by serializing structured features into natural language, but their performance in low-data regimes remains limited compared to gradient-boosted decision trees (GBDTs). In this work, we revisit the boosting paradigm, traditionally associated with tree ensembles, and ask whether it can be applied as a general training principle for LLM fine-tuning. We propose BoostLLM, a framework that transforms parameter-efficient fine-tuning into a multi-round residual optimization process by training sequential PEFT adapters as weak learners. To incorporate tabular inductive bias, BoostLLM integrates decision-tree paths as a second input view alongside raw features; analysis reveals that the path view acts as a structured teacher in early training steps before the model shifts toward feature-driven representations. Empirically, BoostLLM achieves consistent improvements over standard fine-tuning across multiple LLM backbones and datasets, matching or surpassing XGBoost across a wide range of shot counts and outperforming GPT-4o-based methods with a 4B model. We further show that the framework scales: pairing with stronger tree models and extended boosting horizons yields additional gains under appropriate stabilization. These results suggest that boosting can serve as a general training principle for LLM fine-tuning, particularly in low-data regimes for structured data.