DIFFA: Large Language Diffusion Models Can Listen and Understand

2025-07-24

AI Summary

Current audio-language models predominantly adopt autoregressive paradigms, struggling to balance generation quality and inference efficiency; diffusion models remain unexplored for speech understanding. This paper introduces DIFFA—the first diffusion-based large-scale audio-language model tailored for spoken language understanding. Methodologically, DIFFA (1) freezes a pretrained diffusion language model and employs lightweight dual adapters for audio–text cross-modal alignment; (2) proposes a two-stage training paradigm leveraging both ASR alignment supervision and synthetic audio-caption instruction data generated by large language models; and (3) supports bidirectional context modeling and controllable semantic-speech generation. Trained on only 960 hours of real ASR data and 127 hours of synthetic data, DIFFA significantly outperforms leading open-source autoregressive models on MMSU, MMAU, and VoiceBench benchmarks. Results demonstrate the efficacy and scalability of diffusion mechanisms for efficient, high-performance spoken language understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advances in Large language models (LLMs) have shown remarkable capabilities across textual and multimodal domains. In parallel, diffusion-based language models have emerged as a promising alternative to the autoregressive paradigm, offering improved controllability, bidirectional context modeling, and robust generation. However, their application to the audio modality remains underexplored. In this work, we introduce extbf{DIFFA}, the first diffusion-based Large Audio-Language Model designed to perform spoken language understanding. DIFFA integrates a frozen diffusion language model with a lightweight dual-adapter architecture that bridges speech understanding and natural language reasoning. We employ a two-stage training pipeline: first, aligning semantic representations via an ASR objective; then, learning instruction-following abilities through synthetic audio-caption pairs automatically generated by prompting LLMs. Despite being trained on only 960 hours of ASR and 127 hours of synthetic instruction data, DIFFA demonstrates competitive performance on major benchmarks, including MMSU, MMAU, and VoiceBench, outperforming several autoregressive open-source baselines. Our results reveal the potential of diffusion-based language models for efficient and scalable audio understanding, opening a new direction for speech-driven AI. Our code will be available at https://github.com/NKU-HLT/DIFFA.git.

Problem

• Exploring diffusion models for audio understanding tasks
• Bridging speech and language with lightweight dual-adapter
• Enhancing audio-language models with synthetic instruction data

Innovation

• Diffusion-based Large Audio-Language Model
• Dual-adapter bridges speech and language
• Two-stage training with ASR and LLMs