🤖 AI Summary
Existing implicit ensemble methods for tabular data are constrained by rank-1 structures, limiting their ability to enhance predictive diversity and performance. This work proposes LoMETab, the first rank-$r$ implicit ensemble framework that strictly expands the hypothesis space of BatchEnsemble. By parameterizing member weights through low-rank factors, LoMETab introduces two controllable dimensions—adapter rank $r$ and initialization scale $\sigma_{\text{init}}$—enabling fine-grained control over inter-member diversity. The method integrates a rank-$r$ identity residual structure based on the Hadamard product, low-rank decomposition, and end-to-end training, with diversity quantified via KL divergence and decision disagreement. Experiments demonstrate that LoMETab significantly outperforms additive low-rank baselines, with $(r, \sigma_{\text{init}})$ configurations tuning member differences across several orders of magnitude; optimal settings vary by dataset, effectively overcoming current performance bottlenecks.
📝 Abstract
Recent tabular learning benchmarks increasingly show a tight performance cluster rather than a clear hierarchy among leading methods, spanning gradient boosted decision trees, attention-based architectures, and implicit ensembles such as TabM. As benchmark gains plateau, a complementary goal is to understand and control the mechanisms that make simple neural tabular models competitive. We propose LoMETab, a rank-$r$ generalization of multiplicative implicit ensembles. LoMETab lifts the rank-1 BatchEnsemble/TabM modulation to a rank-$r$ identity-residual Hadamard family by parameterizing each member weight as $W_k = W \odot (1 + A_kB_k^\top)$, where $W$ is shared and $(A_k, B_k)$ are member-specific low-rank factors. This exposes two practical diversity-control axes: the adapter rank $r$ and the initialization scale $σ_{\mathrm{init}}$, and we prove that for $r \ge 2$ this generalization strictly enlarges BatchEnsemble's hypothesis class. Empirically, we show that this added capacity manifests as measurable predictive diversity after training: on representative classification datasets, LoMETab sustains higher pairwise KL than an additive low-rank ablation, and $(r, σ_{\mathrm{init}})$ provides broad control over pairwise KL, varying by up to several orders of magnitude across configurations. The induced diversity is reflected in task-appropriate output-level measures: argmax disagreement for classification and ambiguity for regression, indicating that the control extends beyond pairwise KL to decision- and output-level member variation. Finally, experiments sweeping over adapter rank $r$ and initialization scale $σ_{\mathrm{init}}$ reveal that predictive performance is dataset-dependent over the $(r, σ_{\mathrm{init}})$ grid, supporting LoMETab as a controllable family of implicit ensembles rather than a fixed rank-1 construction.