Behavior Knowledge Merge in Reinforced Agentic Models

2026-01-20

AI Summary

Existing model merging methods primarily target supervised fine-tuning and struggle to effectively preserve the distinctive behavioral capabilities of reinforcement learning agents, which arise from sparse and heterogeneous task vectors. To address this limitation, this work proposes the Reinforced Agent Merging (RAM) framework—the first merging approach specifically designed for reinforcement learning agents with distribution-aware mechanisms. RAM decouples shared and task-specific parameters, averaging the shared components while selectively retaining and rescaling the task-specific ones. This strategy successfully integrates behavioral knowledge across multiple agents, consistently outperforming current merging baselines across diverse tasks and model architectures. Notably, the merged models not only maintain performance on individual tasks but often surpass the original expert agents.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Mixture of Experts (MoE)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Reinforcement learning (RL) is central to post-training, particularly for agentic models that require specialized reasoning behaviors. In this setting, model merging offers a practical mechanism for integrating multiple RL-trained agents from different tasks into a single generalist model. However, existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models. The root is a task-vector mismatch between RL and SFT: on-policy RL induces task vectors that are highly sparse and heterogeneous, whereas SFT-style merging implicitly assumes dense and globally comparable task vectors. When standard global averaging is applied under this mismatch, RL's non-overlapping task vectors that encode critical task-specific behaviors are reduced and parameter updates are diluted. To address this issue, we propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models. RAM disentangles shared and task-specific unique parameter updates, averaging shared components while selectively preserving and rescaling unique ones to counteract parameter update dilution. Experiments across multiple agent domains and model architectures demonstrate that RAM not only surpasses merging baselines, but also unlocks synergistic potential among agents to achieve performance superior to that of specialized agents in their domains.

Problem

• model merging
• reinforcement learning
• agentic models
• task-specific capabilities
• parameter dilution

Innovation

• Reinforced Agent Merging
• task-vector mismatch
• distribution-aware merging
• parameter disentanglement
• agentic models