Learning When to Think While Listening in Large Audio-Language Models

2026-05-26

AI Summary

This work addresses the trade-off between response latency and inference quality in streaming spoken-language interaction with large audio language models. The authors propose a learnable three-stage “wait-think-respond” control mechanism that, for the first time, models human-like progressive reasoning in dialogue as a dynamic decision policy. The entire inference trajectory is jointly optimized via a six-dimensional reward function. Built upon the Qwen2.5-Omni-7B model and combining supervised fine-tuning (SFT) with decoupled pruning and dynamic sampling optimization (DAPO), the method improves accuracy on the SRQA benchmark from 67.6% to 70.3% while reducing post-utterance thinking duration by 14%. In evaluations with real human-recorded speech, DAPO emerges as the only learnable controller capable of effectively shortening thinking length.

Abstract

Recent advances in Large Audio-Language Models (LALMs) have made real-time, streaming spoken interaction increasingly practical. In this setting, reasoning quality and responsiveness are tightly coupled: delaying reasoning until the speech endpoint can improve answer quality but moves deliberation into user-visible response delay, while answering too early risks committing before decisive evidence arrives. We introduce a learnable wait-think-answer control formulation for LALMs. Motivated by the incremental nature of human conversation, the controller decides under partial audio evidence when to wait, when to externalize a compact reasoning update, and when to answer. Using Qwen2.5-Omni-7B as the base model, we construct aligned wait-think-answer traces from spoken reasoning data, train the controller with supervised fine-tuning (SFT), and then apply Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO). The reward combines answer correctness, action validity, update timing, latency synchronization, reasoning quality, and chain consistency, optimizing the complete wait-think-answer trajectory and not the final answer alone. On a six-task synthetic spoken reasoning question answering (SRQA) benchmark, the six-reward DAPO controller improves the row-weighted accuracy from 67.6% to 70.3% while reducing post-endpoint final-think length by 14% under the same Qwen deployment harness. On a 186-item human-recorded Real Audio Bench, a transfer check beyond text-to-speech (TTS)-rendered speech, the controller family remains functional: SFT achieves the strongest accuracy, while the six-reward DAPO controller is the only learned variant whose final-think length falls below the base. These results suggest that a streaming model should learn when to make intermediate reasoning explicit during the audio stream.

Problem

• Large Audio-Language Models
• streaming spoken interaction
• reasoning latency
• response delay
• real-time reasoning

Innovation

• wait-think-answer control
• Large Audio-Language Models
• streaming spoken interaction
• DAPO
• incremental reasoning