Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs

2026-04-14

AI Summary

This work addresses the limitations of current audio large language models (LLMs), which, despite strong performance on complex reasoning tasks, exhibit constrained fine-grained acoustic perception due to training paradigms centered predominantly on speech recognition that overlook paralinguistic and non-linguistic acoustic cues. To overcome this, the authors propose the Unified Audio Schema (UAS)—a structured supervision framework that, for the first time, jointly models linguistic, paralinguistic, and non-linguistic audio content within a unified JSON format. The approach integrates both discrete and continuous audio LLM architectures and employs a multi-task training strategy. Evaluated on the MMSU, MMAR, and MMAU benchmarks, the method achieves substantial gains in perceptual performance, improving the MMSU score by 10.9% over same-scale state-of-the-art models while preserving strong reasoning capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent Audio Large Language Models (AudioLLMs) exhibit a striking performance inversion: while excelling at complex reasoning tasks, they consistently underperform on fine-grained acoustic perception. We attribute this gap to a fundamental limitation of ASR-centric training, which provides precise linguistic targets but implicitly teaches models to suppress paralinguistic cues and acoustic events as noise. To address this, we propose Unified Audio Schema (UAS), a holistic and structured supervision framework that organizes audio information into three explicit components -- Transcription, Paralinguistics, and Non-linguistic Events -- within a unified JSON format. This design achieves comprehensive acoustic coverage without sacrificing the tight audio-text alignment that enables reasoning. We validate the effectiveness of this supervision strategy by applying it to both discrete and continuous AudioLLM architectures. Extensive experiments on MMSU, MMAR, and MMAU demonstrate that UAS-Audio yields consistent improvements, boosting fine-grained perception by 10.9% on MMSU over the same-size state-of-the-art models while preserving robust reasoning capabilities. Our code and model are publicly available at https://github.com/Tencent/Unified_Audio_Schema.

Problem

• Audio Large Language Models
• fine-grained acoustic perception
• ASR-centric training
• paralinguistic cues
• non-linguistic events

Innovation

• Unified Audio Schema
• Audio Large Language Models
• Paralinguistics
• Acoustic Perception
• Structured Supervision