Closing the Modality Reasoning Gap for Speech Large Language Models

2026-01-09

arXiv.org

AI Summary

This work addresses the significant performance gap in reasoning capabilities of large speech language models (SLMs) between spoken and textual inputs—a phenomenon termed modality reasoning gap. To bridge this gap, the authors propose the TARS framework, which introduces a novel dual-signal reinforcement learning mechanism that jointly enforces representation alignment and behavioral alignment. By leveraging asymmetric rewards, TARS guides the model to produce consistent reasoning trajectories under both speech and text conditions. Built upon a Transformer-based SLM architecture, the method integrates layer-wise hidden state similarity metrics with semantic consistency evaluation. Experiments on reasoning benchmarks such as MMSU and OBQA demonstrate that TARS substantially narrows the modality gap and achieves state-of-the-art performance on 7B-scale models.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Although speech large language models have achieved notable progress, a substantial modality reasoning gap remains: their reasoning performance on speech inputs is markedly weaker than on text. This gap could be associated with representational drift across Transformer layers and behavior deviations in long-chain reasoning. To address this issue, we introduce TARS, a reinforcement-learning framework that aligns text-conditioned and speech-conditioned trajectories through an asymmetric reward design. The framework employs two dense and complementary signals: representation alignment, which measures layer-wise hidden-state similarity between speech- and text-conditioned trajectories, and behavior alignment, which evaluates semantic consistency between generated outputs and reference text completions. Experiments on challenging reasoning benchmarks, including MMSU and OBQA, show that our approach significantly narrows the modality reasoning gap and achieves state-of-the-art performance among 7B-scale Speech LLMs.

Problem

• modality reasoning gap
• speech large language models
• reasoning performance
• representational drift
• long-chain reasoning

Innovation

• modality reasoning gap
• speech large language models
• reinforcement learning
• representation alignment
• behavior alignment