Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Models

2025-05-30

AI Summary

Existing multimodal large language models (MLLMs) struggle to balance training stability and cross-task generalization in multi-task reinforcement learning, with prevailing methods limited to single-task optimization. Method: We propose Mixed-R1, a unified reward framework featuring four task-specific reward functions—matching, chart understanding, IoU-based localization, and open-ended generation—alongside a bidirectional maximum mean similarity (BMAS) mechanism for open-ended reward estimation. Integrated with the GRPO algorithm, tokenizer embedding alignment, a multi-source data engine for high-quality sample selection, and a hybrid reward weighting strategy, we construct Mixed-45K, a high-fidelity post-training dataset. Contribution/Results: Evaluated on full-parameter Qwen2.5-VL and Intern-VL models, Mixed-R1 achieves significant improvements across diverse tasks—including mathematical reasoning, chart comprehension, visual grounding, and long-text generation—demonstrating superior multi-task capability and generalization. All code, datasets, and models are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent works on large language models (LLMs) have successfully demonstrated the emergence of reasoning capabilities via reinforcement learning (RL). Although recent efforts leverage group relative policy optimization (GRPO) for MLLMs post-training, they constantly explore one specific aspect, such as grounding tasks, math problems, or chart analysis. There are no works that can leverage multi-source MLLM tasks for stable reinforcement learning. In this work, we present a unified perspective to solve this problem. We present Mixed-R1, a unified yet straightforward framework that contains a mixed reward function design (Mixed-Reward) and a mixed post-training dataset (Mixed-45K). We first design a data engine to select high-quality examples to build the Mixed-45K post-training dataset. Then, we present a Mixed-Reward design, which contains various reward functions for various MLLM tasks. In particular, it has four different reward functions: matching reward for binary answer or multiple-choice problems, chart reward for chart-aware datasets, IoU reward for grounding problems, and open-ended reward for long-form text responses such as caption datasets. To handle the various long-form text content, we propose a new open-ended reward named Bidirectional Max-Average Similarity (BMAS) by leveraging tokenizer embedding matching between the generated response and the ground truth. Extensive experiments show the effectiveness of our proposed method on various MLLMs, including Qwen2.5-VL and Intern-VL on various sizes. Our dataset and model are available at https://github.com/xushilin1/mixed-r1.

Problem

• Lack of multi-source MLLM tasks for stable reinforcement learning
• Need unified reward design for diverse MLLM tasks
• Absence of effective open-ended reward for long-form text

Innovation

• Mixed-Reward design for diverse MLLM tasks
• Mixed-45K dataset for stable reinforcement learning
• BMAS reward for long-form text evaluation