Omni-R1: Do You Really Need Audio to Fine-Tune Your Audio LLM?

2025-05-14

AI Summary

This work investigates the necessity of audio modality in fine-tuning audio-capable large language models (LLMs). Building upon the Qwen2.5-Omni multimodal architecture, we apply the GRPO reinforcement learning algorithm to fine-tune the model exclusively on text-only audio question-answering data—without access to raw audio inputs. Our experiments demonstrate that text-only fine-tuning substantially improves performance on audio understanding tasks; the GRPO gains stem primarily from enhanced textual reasoning rather than explicit audio feature modeling, as confirmed via ablation studies with controlled modality inputs. On the MMAU benchmark (under both Test-mini and Test-full splits), our approach establishes new state-of-the-art results across sound, music, speech, and overall average accuracy. This is the first study to empirically verify that high-quality textual instruction data alone can elicit implicit audio comprehension capabilities in multimodal LLMs, offering a novel, low-resource paradigm for audio AI training.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We propose Omni-R1 which fine-tunes a recent multi-modal LLM, Qwen2.5-Omni, on an audio question answering dataset with the reinforcement learning method GRPO. This leads to new State-of-the-Art performance on the recent MMAU benchmark. Omni-R1 achieves the highest accuracies on the sounds, music, speech, and overall average categories, both on the Test-mini and Test-full splits. To understand the performance improvement, we tested models both with and without audio and found that much of the performance improvement from GRPO could be attributed to better text-based reasoning. We also made a surprising discovery that fine-tuning without audio on a text-only dataset was effective at improving the audio-based performance.

Problem

• Fine-tuning audio LLMs without using audio data
• Improving audio question answering via text-based reasoning
• Achieving state-of-the-art performance on MMAU benchmark

Innovation

• Uses GRPO reinforcement learning for fine-tuning
• Achieves SOTA on MMAU benchmark without audio
• Text-only fine-tuning boosts audio performance