Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging

2026-06-16

AI Summary

This work addresses the significant performance degradation observed in existing model merging techniques when integrating multiple independently trained reinforcement learning with verifiable rewards (RLVR) models. The authors identify the root cause as a “curse of sparsity,” stemming from the highly sparse and spatially scattered parameter updates characteristic of RLVR models. To mitigate this, they propose SAR-Merging, the first merging method specifically designed for RLVR, which leverages Fisher information to assess parameter sensitivity and incorporates conflict resolution among updated regions, magnitude-aware sparsification, and rescaling mechanisms to preserve critical reasoning pathways. Experimental results demonstrate that SAR-Merging substantially outperforms current merging strategies on mathematical and code reasoning benchmarks, achieving both enhanced single-task performance and effective multi-ability integration.

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful post-training paradigm that surpasses Supervised Fine-Tuning (SFT) in eliciting reasoning intelligence and resisting catastrophic forgetting. Recent studies further reveal that RLVR induces highly sparse and off-principal parameter updates compared to SFT. This naturally raises the question: does such sparsity make RLVR models more amenable to model merging? If so, model merging would offer a scalable, training-free path to aggregate diverse reasoning capabilities from independently trained RLVR models. Surprisingly, we find the opposite, uncovering a sparsity curse: the sparse RLVR updates are spread farther apart in parameter space, forming near-orthogonal shortcuts that make aggregation inherently fragile. This is likely rooted in the stochasticity of RL optimization and the diversity of emergent reasoning patterns. Unlike SFT models that converge to shared, flat basins and merge naturally, RLVR models suffer severe degradation under standard merging methods. Through systematic empirical analysis of the update geometry, we characterize the mechanisms behind this failure and propose Sensitivity-aware Resolving Merging (SAR-Merging), a merging recipe tailored for the unique structure of RLVR parameter spaces. SAR-Merging resolves conflicts in overlapping update regions via Fisher Information-based sensitivity arbitration, followed by magnitude-aware sparsification and rescaling to preserve fragile reasoning pathways. Experiments on mathematical and coding benchmarks demonstrate that SAR-Merging substantially outperforms existing merging methods on RLVR models, enabling both single-task enhancement and multi-capability fusion.

Problem

• RLVR
• model merging
• sparsity curse
• parameter space
• catastrophic forgetting

Innovation

• sparsity curse
• RLVR
• model merging
• SAR-Merging
• parameter space geometry