Reinforcement Fine-Tuning Enables MLLMs Learning Novel Tasks Stably

2025-06-30

AI Summary

This study investigates the trade-off between prior knowledge retention and task adaptation in multimodal large language models (MLLMs), comparing supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). To systematically evaluate stability and forgetting, we propose a novel downstream “puzzle task.” Experiments reveal that although RFT converges more slowly, it significantly mitigates catastrophic forgetting; crucially, its rollout sampling trajectories implicitly encode knowledge retention signals. Leveraging this insight, we design a trajectory-guided SFT strategy, achieving synergistic optimization of rapid adaptation and knowledge preservation on Qwen2.5-VL. A key finding is that data distribution shift—not parameter update magnitude—is the primary driver of forgetting, and RFT inherently alleviates this shift. Our work establishes an interpretable, transferable fine-tuning paradigm for MLLM continual learning, bridging theoretical insight with practical efficacy.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Multiagent Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Post-training algorithms such as Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) are widely used to adapt multimodal large language models to downstream tasks. While effective at task adaptation, their impact on prior knowledge remains unclear. In this paper, we introduce jigsaw puzzles as a novel task absent from existing pretraining corpora and systematically study the behavior of SFT and RFT on an open-source multimodal model, Qwen2.5-VL. Our experiments reveal a sharp trade-off: SFT enables rapid task acquisition but leads to catastrophic forgetting, whereas RFT learns more slowly on novel tasks but maintains prior knowledge. We analyze this phenomenon through the lens of learning dynamics, showing that RFT reinforces correct samples that are naturally aligned with the base model's probability landscape, mitigating interference with prior knowledge. Moreover, supervised training on correct RFT-simulated rollouts allows SFT to preserve knowledge while rapidly learning new tasks. These findings suggest that data distribution, rather than algorithmic differences, plays a central role in forgetting, and highlight RFT's potential for stable continual learning in multimodal large language models.

Problem

• Study trade-off between task learning and knowledge retention in MLLMs
• Compare SFT and RFT impacts on novel task adaptation
• Analyze data distribution role in catastrophic forgetting

Innovation

• Reinforcement Fine-Tuning stabilizes novel task learning
• RFT mitigates forgetting by aligning with base model
• Supervised training on RFT rollouts preserves prior knowledge