Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition

2026-07-09

AI Summary

This work addresses the inefficiency of cross-modal fusion caused by representational discrepancies among audio, visual, and textual modalities by proposing a semantic alignment method based on optimal transport (OT). For the first time, OT is integrated into a large language model–driven audio-visual speech recognition (LLM-AVSR) framework. Leveraging LLaMA3.2-3B language embeddings as anchors, the approach employs OT-derived coupling matrices to generate soft pseudo-labels that guide contrastive learning, thereby explicitly aligning features from Whisper’s audio encoder and AV-HuBERT’s visual encoder in a language-anchored semantic space. Evaluated on the LRS3-TED benchmark, the method significantly outperforms strong existing baselines and achieves state-of-the-art performance across varying signal-to-noise ratios, substantially enhancing recognition robustness and semantic consistency in challenging acoustic environments.

Abstract

Large language model (LLM)-based audio-visual speech recognition (LLM-AVSR) has recently demonstrated strong robustness in adverse acoustic environments by leveraging complementary audio and visual information. Existing approaches typically employ independently pretrained acoustic and visual encoders, whose outputs are projected and fused as soft prompts to condition an LLM for speech recognition. However, most methods perform multimodal fusion without explicitly addressing the representational discrepancy between audio, visual and text modalities, potentially limiting the effectiveness of cross-modal integration. In this paper, we propose an optimal transport (OT)-based semantic alignment framework for LLM-AVSR. The proposed method explicitly bridges the modality gap by aligning the acoustic and visual representations with reference to the linguistic embedding space of the LLM before multimodal fusion. Specifically, OT is used to estimate probabilistic coupling matrices that characterize structured correspondences between modality-specific features and linguistic embeddings. The resulting OT couplings are further utilized as soft pseudo-labels to supervise contrastive learning, encouraging the extraction of semantically coherent and cross-modal consistent audio-visual representations. By anchoring multimodal features to the linguistic space of the LLM, the proposed framework facilitates more effective multimodal fusion and decoding. We implement the proposed framework using a Whisper-based acoustic encoder, an AV-HuBERT-based visual encoder, and a LLaMA3.2-3B decoder. Experiments conducted on the LRS3-TED benchmark demonstrate consistent improvements over strong baselines and achieve state-of-the-art performance under both clean and noisy evaluation conditions across a wide range of signal-to-noise ratios (SNRs).

Problem

• audio-visual speech recognition
• multimodal fusion
• representational discrepancy
• cross-modal integration
• large language model

Innovation

• Optimal Transport
• Semantic Alignment
• Multimodal Fusion
• LLM-based AVSR
• Contrastive Learning