Linear Mode Connectivity in Differentiable Tree Ensembles

2024-05-23

arXiv.org

AI Summary

This work addresses the challenge of achieving stable linear mode connectivity (LMC) in differentiable tree ensemble models—a property long observed in neural networks but elusive in tree-based architectures. Unlike neural networks, where neuron permutation symmetry facilitates LMC, we identify subtree flipping and split-order invariance as fundamental obstructions to LMC in trees. To overcome this, we propose a novel decision-list-based tree architecture that intrinsically avoids these invariances, thereby enabling structural LMC guarantees. Our method integrates invariance-aware modeling, decision-list parameterization, and optimization-path connectivity verification. Empirical evaluation across multiple benchmark datasets demonstrates that linear interpolations between independently trained models incur less than 1% performance degradation—without requiring any symmetry alignment procedures. This substantially enhances the feasibility and robustness of parameter-space operations such as model merging, marking the first successful realization of stable LMC in differentiable tree ensembles.

Technology Category

• Machine Learning: Ensemble Methods
• Search and Optimization: Mixed Discrete/Continuous Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Linear Mode Connectivity (LMC) refers to the phenomenon that performance remains consistent for linearly interpolated models in the parameter space. For independently optimized model pairs from different random initializations, achieving LMC is considered crucial for understanding the stable success of the non-convex optimization in modern machine learning models and for facilitating practical parameter-based operations such as model merging. While LMC has been achieved for neural networks by considering the permutation invariance of neurons in each hidden layer, its attainment for other models remains an open question. In this paper, we first achieve LMC for soft tree ensembles, which are tree-based differentiable models extensively used in practice. We show the necessity of incorporating two invariances: subtree flip invariance and splitting order invariance, which do not exist in neural networks but are inherent to tree architectures, in addition to permutation invariance of trees. Moreover, we demonstrate that it is even possible to exclude such additional invariances while keeping LMC by designing decision list-based tree architectures, where such invariances do not exist by definition. Our findings indicate the significance of accounting for architecture-specific invariances in achieving LMC.

Problem

• Achieving Linear Mode Connectivity in tree ensembles
• Incorporating subtree flip and splitting order invariances
• Designing decision list-based tree architectures for LMC

Innovation

• Achieves LMC in tree ensembles
• Incorporates subtree flip invariance
• Designs decision list-based architectures