Quantifying Cross-Lingual Transfer in Paralinguistic Speech Tasks

2026-03-09

AI Summary

This study addresses the lack of systematic quantification of cross-lingual transfer effects in paralinguistic speech tasks, which has hindered the assessment of linguistic dependency across language pairs and tasks. To this end, we propose the Cross-Lingual Transfer Matrix (CLTM) framework, which enables the first fine-grained and highly comparable quantitative analysis of transfer effects in paralinguistic tasks. Leveraging a multilingual HuBERT encoder, we construct transfer performance matrices across language pairs for gender identification and speaker verification. Our experiments reveal significant and systematic cross-lingual transfer patterns, demonstrating that paralinguistic tasks exhibit non-negligible linguistic dependency. These findings offer a novel perspective for multilingual speech modeling and underscore the importance of accounting for language-specific characteristics even in ostensibly language-agnostic paralinguistic tasks.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Paralinguistic speech tasks are often considered relatively language-agnostic, as they rely on extralinguistic acoustic cues rather than lexical content. However, prior studies report performance degradation under cross-lingual conditions, indicating non-negligible language dependence. Still, these studies typically focus on isolated language pairs or task-specific settings, limiting comparability and preventing a systematic assessment of task-level language dependence. We introduce the Cross-Lingual Transfer Matrix (CLTM), a systematic method to quantify cross-lingual interactions between pairs of languages within a given task. We apply the CLTM to two paralinguistic tasks, gender identification and speaker verification, using a multilingual HuBERT-based encoder, to analyze how donor-language data affects target-language performance during fine-tuning. Our results reveal distinct transfer patterns across tasks and languages, reflecting systematic, language-dependent effects.

Problem

• cross-lingual transfer
• paralinguistic speech tasks
• language dependence
• systematic assessment

Innovation

• Cross-Lingual Transfer Matrix
• paralinguistic speech tasks
• language dependence
• multilingual HuBERT
• systematic evaluation