ResMerge: Residual-based Spectral Merging of Large Language Models

2026-06-01

AI Summary

Existing spectral merging methods struggle to effectively fuse untrained expert models in reinforcement learning (RL) due to invalid assumptions about the structure of task vectors, often resulting in capability loss or interference. This work proposes ResMerge, a novel framework that reveals— for the first time—that the residual components within RL task vectors encode rich and stable behavioral knowledge, making them more suitable for aggregation. ResMerge decomposes task vectors into spectral heads and residual parts, constructs a shared backbone via spherical residual consensus, and incorporates a lightweight gated head-correction module to integrate higher-order information. Experiments demonstrate that ResMerge significantly outperforms current task-vector and spectral merging baselines across multiple RL expert models and capability domains, achieving superior preservation of expert capabilities.

Abstract

Model merging offers a training-free way to combine multiple post-trained expert models, but merging experts obtained through reinforcement learning (RL) remains challenging. Existing spectral merging methods often assume that leading singular directions contain the main task signal, while lower-energy residual components can be compressed, selected, or attenuated to reduce interference. We find that this assumption does not hold for RL task vectors: after decomposing each task vector into a leading spectral head and a residual component, both parts can independently recover substantial behavior knowledge, while exhibiting different merging properties. The head is highly concentrated and informative but more prone to sharp cross-expert conflicts, whereas the residual component is more dispersed and provides a more stable basis for aggregation. Based on this observation, we propose ResMerge, a residual-based spectral merging framework for RL experts. ResMerge first constructs a stable residual backbone with Spherical Residual Consensus Adaptation, which estimates a reliability-weighted consensus direction on the Frobenius sphere. It then reintroduces leading-head information through a Lightweight Head Correction module gated by positive cross-expert agreement. Experiments across multiple RL expert groups and capability domains show that ResMerge better preserves expert capabilities than representative task-vector and spectral merging baselines. The implementation of ResMerge is publicly available at https://github.com/sunyd0303-cpu/ResMerge-release.

Problem

• model merging
• reinforcement learning
• spectral merging
• task vectors
• large language models

Innovation

• ResMerge
• spectral merging
• reinforcement learning
• residual component
• task vector