AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers

2026-04-17

AI Summary

This work addresses the scarcity of high-quality multimodal reasoning data, which hinders the effective transfer of textual reasoning capabilities to audio-visual domains. The authors propose a novel cross-modal transfer paradigm that operates without original multimodal reasoning data: unimodal teacher models first generate separate visual and auditory reasoning traces, which are then fused by a large language model to synthesize multimodal reasoning samples. These synthetic samples are leveraged in a staged training pipeline combining supervised fine-tuning and reinforcement learning. The approach not only yields high-quality training data but also demonstrates positive transfer from multimodal training back to unimodal tasks. Evaluated on seven audio-visual benchmarks—including OmniBench, DailyOmni, and MMAR—the resulting model achieves state-of-the-art performance among models of comparable scale, establishing a new standard for multimodal reasoning training.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advances in reasoning models have shown remarkable progress in text-based domains, but transferring those capabilities to multimodal settings, e.g., to allow reasoning over audio-visual data, still remains a challenge, in part because of the limited availability of high-quality reasoning data in targeted multimodal combinations. To address this problem, we introduce AVRT, a novel framework that generates high-quality audio-visual reasoning traces from single-modality teacher models. We generate independent vision- and audio-reasoning traces via models specialized to reason over their respective modalities and merge the resulting traces with an LLM merger model. The resulting multimodal traces are used in a supervised fine-tuning (SFT) cold start to adapt the target model to audio-visual reasoning traces first, before training it in a second reinforcement learning stage on larger-scale data. Evaluated on seven audio-visual and audio benchmarks, our 3B and 7B parameter models achieve state-of-the-art results among models of comparable size including OmniBench and DailyOmni for audio-visual and MMAR for audio-only reasoning, showing that cross-modal training also transfers to single-modality tasks and establishing a new training pipeline for multimodal reasoning models.

Problem

• multimodal reasoning
• audio-visual reasoning
• reasoning transfer
• data scarcity
• cross-modal learning

Innovation

• audio-visual reasoning
• single-modality teachers
• reasoning trace generation
• multimodal fusion
• two-stage training