How Bilingual Are SSL Speech Models? Cross-Lingual Probing of Articulatory Encoding with Finnish and Russian EMA

2026-06-30

AI Summary

This study investigates how self-supervised speech (SSL) models encode articulatory movement information across languages, focusing on Finnish–Russian bilingual speakers. Leveraging real bilingual electromagnetic articulography (EMA) data, the authors systematically evaluate the ability of SSL model hidden representations to predict articulator movements through cross-lingual linear probing and Pearson correlation analysis. For the first time, the cross-lingual articulatory encoding capacity of SSL models is validated on authentic bilingual EMA data, revealing that intermediate layers yield optimal articulatory feature representations and that tongue movements are more predictable than lip movements. Using only approximately five minutes of training data, multilingual SSL models achieve a peak Pearson correlation coefficient of 0.68 on read-speech tasks—significantly outperforming monolingual counterparts—and demonstrate robust generalization across varying levels of language proficiency.

Abstract

SSL speech models capture rich phonetic, prosodic, and acoustic patterns from raw audio, yet how they encode articulatory information across diverse languages remains unclear. Using EMA data from bilingual Finnish-Russian speakers, we evaluate cross-lingual correlations between SSL latent representations and articulatory movements. Models achieve strong prediction performance (Pearson r up to 0.68) even with approximately 5 minutes of training data, with multilingual models outperforming monolingual ones. Intermediate layers encode articulatory features most effectively, and tongue movements are more predictable than lip movements. We also assess the impact of task type (read versus spontaneous speech) and language proficiency, finding higher accuracy for structured tasks and strong generalization across proficiency levels. These results enhance the interpretability of SSL models and show their potential for speech-technology applications.

Problem

• SSL speech models
• articulatory encoding
• cross-lingual
• bilingual
• EMA

Innovation

• self-supervised learning
• cross-lingual probing
• articulatory encoding
• EMA
• multilingual speech models