NAVER LABS Europe Submission to the Instruction-following 2026 Short Track

2026-07-02

AI Summary

This work proposes a compact multitask system that jointly performs automatic speech recognition (ASR), speech translation (ST), and spoken question answering (SQA) from English speech into Chinese, Italian, and German. The approach introduces SpeechMapper to replace conventional speech projectors, effectively mapping speech representations into the embedding space of a large language model (LLM). Multistage training leverages synthetic speech from SeamlessM4T-large-v2 and a domain-specialized dataset, fakACL, generated by the LLM. Evaluated in the IWSLT 2026 short track, the system ties for first place with the top-performing submission, outperforming last year’s champion while employing a smaller model architecture and reduced reliance on LLM capabilities. This advancement significantly enhances the efficiency and domain-specific proficiency of cross-lingual speech understanding and generation under resource-constrained conditions.

Abstract

In this paper, we describe NAVER LABS Europe's submission to the instruction-following speech processing short track at IWSLT 2026. We participate again in the constrained setting, developing systems capable of jointly performing ASR, ST, and SQA from English speech into Chinese, Italian, and German. Building on our previous submission, ranked first in last year's short track, we update our multi-stage training pipeline by replacing the speech projector with SpeechMapper, a method for learning a speech-to-LLM embedding projector using only ASR data. In addition, we introduce a synthetic SQA dataset, fakACL, composed of artificially generated scientific presentations. This dataset is built by prompting the LLM backbone, segmenting the generated talks, and synthesizing speech with SeamlessM4T-large-v2. The combination of an improved speech projection mechanism and domain-specific synthetic data allows our model to outperform last year's best short-track system, while being considerably more compact and relying on a weaker LLM backbone. This year's results place our system tied for first place in the overall short track ranking.

Problem

• instruction-following
• speech processing
• ASR
• ST
• SQA

Innovation

• SpeechMapper
• synthetic SQA dataset
• speech-to-LLM projection
• multi-stage training pipeline
• instruction-following speech processing