video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model

2025-02-17

AI Summary

Existing video understanding research predominantly focuses on mathematical reasoning or unimodal visual processing, neglecting general-purpose audio-visual joint reasoning. This work introduces the first large multimodal model explicitly designed for general video understanding and enhanced with structured reasoning capabilities, overcoming modality and task limitations. Methodologically, we propose process-oriented Direct Preference Optimization (pDPO), integrating step-level multimodal reward modeling with contrastive step selection. We further construct RivaBench—the first high-quality audio-visual reasoning benchmark—and demonstrate zero-shot synthetic video detection. Experiments show that our model achieves 3–8% absolute accuracy gains over LLaVA-OneVision across multiple video reasoning benchmarks; pDPO yields 6–8% improvements over supervised fine-tuning on RivaBench. These results significantly advance large-model capabilities in audio-visual co-reasoning.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While recent advancements in reasoning optimization have significantly enhanced the capabilities of large language models (LLMs), existing efforts to improve reasoning have been limited to solving mathematical problems and focusing on visual graphical inputs, neglecting broader applications in general video understanding.This paper proposes video-SALMONN-o1, the first open-source reasoning-enhanced audio-visual LLM designed for general video understanding tasks. To enhance its reasoning abilities, we develop a reasoning-intensive dataset featuring challenging audio-visual questions with step-by-step solutions. We also propose process direct preference optimization (pDPO), which leverages contrastive step selection to achieve efficient step-level reward modelling tailored for multimodal inputs. Additionally, we introduce RivaBench, the first reasoning-intensive video understanding benchmark, featuring over 4,000 high-quality, expert-curated question-answer pairs across scenarios such as standup comedy, academic presentations, and synthetic video detection. video-SALMONN-o1 achieves 3-8% accuracy improvements over the LLaVA-OneVision baseline across different video reasoning benchmarks. Besides, pDPO achieves 6-8% improvements compared to the supervised fine-tuning model on RivaBench. Enhanced reasoning enables video-SALMONN-o1 zero-shot synthetic video detection capabilities.

Problem

• Enhance general video understanding
• Develop reasoning-intensive audio-visual dataset
• Introduce multimodal step-level reward modeling

Innovation

• Open-source audio-visual LLM
• Process direct preference optimization
• Reasoning-intensive video benchmark