EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning

2025-05-07

AI Summary

Multimodal large language models (MLLMs) exhibit insufficient cross-modal understanding in synchronous audio-visual reasoning and multiple-choice question answering. Method: We propose EchoInk-R1-7B, a reinforcement learning–based structured reasoning framework. It introduces the lightweight Group Relative Policy Optimization (GRPO) algorithm to unify audio, visual, and textual representations; incorporates a reflective reasoning mechanism for iterative response refinement under ambiguous inputs; and establishes AVQA-R1-6K—a novel synchronized audio-visual multiple-choice QA benchmark comprising 6K samples. Contribution/Results: On AVQA, EchoInk-R1-7B achieves 85.77% accuracy—outperforming baselines by 5.24 percentage points—with only 562 RL training steps. This work significantly enhances MLLMs’ fine-grained parsing and robust reasoning capabilities over ambiguous audio-visual inputs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multimodal large language models (MLLMs) have advanced perception across text, vision, and audio, yet they often struggle with structured cross-modal reasoning, particularly when integrating audio and visual signals. We introduce EchoInk-R1, a reinforcement learning framework that enhances such reasoning in MLLMs. Built upon the Qwen2.5-Omni-7B foundation and optimized with Group Relative Policy Optimization (GRPO), EchoInk-R1 tackles multiple-choice question answering over synchronized audio-image pairs. To enable this, we curate AVQA-R1-6K, a dataset pairing such audio-image inputs with multiple-choice questions derived from OmniInstruct-v1. EchoInk-R1-7B achieves 85.77% accuracy on the validation set, outperforming the base model, which scores 80.53%, using only 562 reinforcement learning steps. Beyond accuracy, EchoInk-R1 demonstrates reflective reasoning by revisiting initial interpretations and refining responses when facing ambiguous multimodal inputs. These results suggest that lightweight reinforcement learning fine-tuning enhances cross-modal reasoning in MLLMs. EchoInk-R1 is the first framework to unify audio, visual, and textual modalities for general open-world reasoning via reinforcement learning. Code and data are publicly released to facilitate further research.

Problem

• Enhancing audio-visual reasoning in multimodal LLMs
• Improving cross-modal integration via reinforcement learning
• Addressing structured reasoning with synchronized audio-image inputs

Innovation

• Reinforcement learning enhances cross-modal reasoning
• GRPO optimizes audio-visual-textual multimodal integration
• Lightweight fine-tuning boosts MLLM accuracy significantly