Connecting Speech to Words through Images

2026-06-15

AI Summary

This study addresses the challenge of aligning spoken words with their written forms in the absence of explicit textual supervision. To this end, the authors propose a method that integrates image captioning, unsupervised spoken word discovery, and speech–image alignment to construct a visually grounded semantic lexicon from images paired with spoken descriptions, and to align speech segments with orthographic word forms in a fully unsupervised manner. Requiring no textual annotations and offering strong interpretability, the approach presents a novel pathway for speech–text alignment in low-resource languages. Experimental results demonstrate that the method significantly outperforms strong neural baselines on English spoken term retrieval and keyword spotting tasks, confirming its effectiveness and practical utility.

Abstract

How can we learn the mapping between written words and their spoken counterparts in the absence of explicit textual supervision? We present a visually grounded method for building a vocabulary of spoken words using only images and their spoken descriptions. First, image captioning systems are used to build a vocabulary of written words representing salient visual concepts in the images. For each word, we then find utterances whose image captions contain that word. Then we use an unsupervised word discovery technique to align these utterances to locate instances of the target word. The result is spoken word segments that are linked to written words -- all accomplished without any text supervision. In spoken word retrieval and keyword spotting experiments, the proposed approach outperforms a strong neural baseline while being more interpretable. These results demonstrate the feasibility of the approach in English and motivate future work on low-resource languages without transcripts.

Problem

• spoken word discovery
• visual grounding
• text-free learning
• word-audio alignment
• low-resource languages

Innovation

• visually grounded
• unsupervised word discovery
• spoken word segmentation
• text-free learning
• keyword spotting