Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers

2026-06-22

AI Summary

Existing linear mode connectivity approaches struggle to scale to large pretrained Transformers due to their reliance on unidirectional optimization of interpolation paths, which limits merging performance. This work proposes a scalable framework that aligns equivalent solutions through function-preserving weight transformations and introduces a bidirectional joint optimization strategy, enabling two models to cooperatively converge toward a shared linear path. The method achieves, for the first time, near-zero loss barrier linear connectivity in billion-parameter-scale Transformers, overcoming a key bottleneck in large-model linear connectability. Experiments demonstrate near-zero loss barriers for medium-sized language models on WikiText, sustained Top-1 accuracy above 69% throughout interpolation for ViT-L on ImageNet, and only minimal loss barriers even in billion-parameter models.

Abstract

Linear mode connectivity (LMC) provides a promising foundation for understanding and merging independently trained neural networks, but existing methods typically optimize the interpolation path from only one model endpoint, limiting their scalability and effectiveness for large pretrained transformers. We propose a novel and scalable framework for enabling LMC-based model merging to {\em billion-parameter pretrained transformers}. Our method applies properly parameterized functionality-preserving weight transformations to align functionally equivalent solutions, and introduces a dual learning procedure in which both models jointly learn their corresponding transformations toward a shared linear interpolation path. This bidirectional optimization substantially reduces interpolation barriers and enables more reliable merging across large-scale architectures. Empirically, we show that our approach achieves near-zero loss barriers on WikiText for language models with medium-sized parameters, representing, to our knowledge, the first demonstration of near-barrier-free linear connectivity at this scale. In the vision domain, ViT-L maintains above 69\% ImageNet top-1 accuracy throughout the interpolation path, while modern billion-parameter LLMs exhibit only small loss barriers. These results suggest that properly resolving parameter symmetries enables large pretrained Transformers to be connected and merged through simple linear paths with substantially improved interpolation performance. Code: https://github.com/VILA-Lab/Dual-Learned-Matching .

Problem

• Linear Mode Connectivity
• Model Merging
• Pretrained Transformers
• Parameter Symmetries
• Scalability

Innovation

• Linear Mode Connectivity
• Model Merging
• Functionality-Preserving Transformation
• Dual Learning
• Parameter Symmetry