AgentPatch: Coarse-to-Fine Weak-Task Repair for Merging Agentic Multimodal Large Language Models

2026-08-06

AI Summary

This work addresses performance degradation on weak tasks and forgetting of critical behaviors in merging multimodal large language models for agents. The authors propose a training-free, coarse-to-fine repair framework that integrates capabilities within a single static model through stable backbone selection, Weak-Task Unique Residual Recovery, and Agent-Guided Behavior-Critical Patching. For the first time, this approach introduces a synergistic mechanism that jointly repairs weak-task performance and preserves behavior-critical knowledge, eliminating the need for dynamic routing or model ensembles. Experimental results across six agent and multimodal benchmarks demonstrate that the method substantially mitigates weak-task degradation while effectively retaining complementary capabilities in visual processing and search.

Abstract

Agentic multimodal large language models (MLLMs) extend multimodal perception and reasoning with planning, tool use, and interaction in dynamic environments. Yet current models are specialized for particular tools or environments, complicating consolidation into a single generalist. We formulate Agentic MLLM Merging and identify two challenges: asymmetric capability preservation, whereby capabilities with different interaction complexity are retained unevenly, producing weak tasks after merging, and behavior-critical forgetting, whereby losing decisive actions can derail long-horizon execution. We propose AgentPatch, a training-free coarse-to-fine repair framework. It selects a stable merged backbone, restores diluted weak-task-specific signals through Weak-Task Unique Residual Recovery, and applies an Agent-Guided Behavior-Critical Patch that recovers decisive behaviors under explicit capability protection. AgentPatch produces a single static checkpoint without routing or ensembles. Experiments across six agentic and multimodal benchmarks show that AgentPatch improves diverse merged backbones, alleviates weak-task degradation, and better balances weak-task recovery with the preservation of complementary search and agentic visual processing capabilities. Code is available at https://github.com/ziboshao/AgentPatch.

Problem

• Agentic MLLM Merging
• asymmetric capability preservation
• behavior-critical forgetting
• weak-task degradation
• multimodal large language models

Innovation

• AgentPatch
• Agentic MLLM Merging
• Weak-Task Repair
• Behavior-Critical Forgetting
• Training-Free Merging