Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation

2026-09-13

AI Summary

为解决多模态指令跟随数据稀缺问题，提出MIFS系统生成RL就绪的多模态数据，通过合成多样化样本和基于学习动态的数据提炼机制，提高模型性能。

Abstract

Multimodal instruction following (MMIF) is crucial for building generalist agents. However, current training paradigms rely heavily on Supervised Fine-Tuning (SFT), which often leads to surface-level pattern matching and degrades general capabilities. While Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising alternative, its scalability in MMIF is severely bottlenecked by the scarcity of high-quality, RL-ready multimodal data. To bridge this gap, we present MIFS (\textbf{M}ultimodal \textbf{I}nstruction \textbf{F}ollowing \textbf{S}ynthesis), a systematic pipeline designed to generate RL-ready multimodal data. Specifically, MIFS introduces a generative constraint protocol to synthesize diverse raw samples, followed by a learnability-aware distillation mechanism that filters data based on RL training dynamics to ensure stable policy optimization. Furthermore, a code-based verifier provides high-precision reward signals for policy learning. The resulting dataset comprises 90k samples across 8 constraint categories and 14 task domains. Empirical evaluations demonstrate that MIFS-trained MLLMs achieve an average improvement of 8.13\% on four MMIF benchmarks and a 3$\times$ faster training convergence compared to using raw data. Crucially, our approach mitigates the generalization trade-offs typical of SFT, preserving core visual capabilities while significantly boosting instruction-following precision.

Problem

• Multimodal Instruction Following
• Supervised Fine-Tuning
• Reinforcement Learning with Verifiable Rewards
• Scalability
• Data Scarcity

Innovation

• Multimodal Instruction Following
• Data Synthesis
• Reinforcement Learning
• Learnability-aware Distillation
• Generative Constraint Protocol