FBK's Long-form SpeechLLMs for IWSLT 2026 Instruction Following

2026-06-25

AI Summary

This work addresses the challenges of generation instability and hallucination in long-form spoken instruction-following tasks by proposing the SpeechLLM framework, which systematically investigates three speech segmentation strategies to enhance robustness in long speech understanding and generation. The study introduces the HIFS metric to quantitatively assess generation stability and finds that fixed 30-second segmentation yields optimal performance. It further reveals that hallucinations predominantly manifest as repetitive insertions, which degrade ASR and summarization quality yet leave short-speech capabilities largely intact. Experimental results demonstrate that the proposed approach achieves a SIFS score of 2.0708 on short-speech tasks and attains the highest HIFS score of 2.0663 on long-speech tasks using 30-second segments, confirming its effectiveness and robustness.

Abstract

This paper describes our submission to the IWSLT 2026 Instruction Following shared task. SpeechLLMs are developed for both short-form and long-form speech instruction following under constrained settings. For the short track, strong performance is achieved on MCIF, with a SIFS score of 2.0708. For the long track, three speech segmentation methods are explored, and the HIFS score is introduced to account for unstable long-form generation. Experimental results show that fixed 30-second segmentation provides the most robust long-form performance, achieving the highest HIFS score of 2.0663. Further analysis shows that hallucination mainly manifests as repetitive insertions in generated outputs, substantially affecting ASR and SSUM, while short-form capabilities are largely retained after long-form extension.

Problem

• SpeechLLMs
• instruction following
• long-form speech
• hallucination
• speech segmentation

Innovation

• SpeechLLM
• long-form speech
• instruction following
• HIFS
• speech segmentation