From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decoding

2026-06-15

AI Summary

This work addresses the limited ability of existing end-to-end spoken dialogue systems to effectively leverage historical context during decoding, which often results in insufficient coherence and faithfulness in multi-turn conversations. To bridge the gap between contextual awareness and actual adherence during generation, the authors propose Context-Aware Decoding with Audio Adaptation (CAD), a novel inference-time approach that dynamically enhances reliance on salient multimodal context by contrasting output distributions with and without critical historical utterances. CAD identifies relevant prior turns through an internal attention mechanism and integrates a contrastive decoding strategy to explicitly align generated responses with essential context. Evaluated on the Audio MultiChallenge benchmark, the method significantly improves semantic memory retention and self-consistency, yielding dialogue responses that more rigorously adhere to contextual constraints.

Abstract

Despite the success of end-to-end (E2E) spoken dialogue systems, maintaining strict context adherence in multi-round conversations remains a challenge. While prior works attribute these failures to models forgetting dialogue history, we highlight an equally critical but overlooked bottleneck: a gap between latent context awareness and active adherence. Although models internally recognize relevant past utterances, strong parametric priors often overshadow these signals during decoding. To bridge this gap, we propose an audio-adapted Context-Aware Decoding (CAD) approach. By leveraging internal attention mechanisms to isolate key historical rounds, our approach contrasts output distributions with and without this key context during inference, directly amplifying multimodal contextual signals. Evaluations on the Audio MultiChallenge benchmark demonstrate significant improvements in Semantic Memory and Self Coherence subtasks, successfully enforcing strict, context-faithful adherence.

Problem

• spoken dialogue systems
• context adherence
• context awareness
• multi-round conversations
• decoding

Innovation

• Context-Aware Decoding
• spoken dialogue systems
• context adherence
• multimodal contextual signals
• audio-adapted inference