LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning
Existing implicit ensemble methods for tabular data are constrained by rank-1 structures, limiting their ability to enhance predictive diversity and performance. This work proposes LoMETab, the first rank-$r$ implicit ensemble framework that strictly expands the hypothesis space of BatchEnsemble. By parameterizing member weights through low-rank factors, LoMETab introduces two controllable dimensions—adapter rank $r$ and initialization scale $\sigma_{\text{init}}$—enabling fine-grained control over inter-member diversity. The method integrates a rank-$r$ identity residual structure based on the Hadamard product, low-rank decomposition, and end-to-end training, with diversity quantified via KL divergence and decision disagreement. Experiments demonstrate that LoMETab significantly outperforms additive low-rank baselines, with $(r, \sigma_{\text{init}})$ configurations tuning member differences across several orders of magnitude; optimal settings vary by dataset, effectively overcoming current performance bottlenecks.