Problem
Research questions and friction points this paper is trying to address.
acoustic-to-articulatory inversion
electromagnetic articulography
data augmentation
articulatory data scarcity
speech-driven animation
ArtBoost: Synthetic Articulatory Data Augmentation for Acoustic-to-Articulatory Inversion
🤖 AI Summary
This work addresses the challenge of articulatory-acoustic inversion (AAI), which is hindered by the high cost and limited scale of electromagnetic articulography (EMA) data. The authors propose ArtBoost, a novel method that leverages large-scale speech–3D facial mesh datasets to generate scalable pseudo-articulatory trajectories as cross-modal supervisory signals. A two-stage learning framework is introduced: the AAI model is first pretrained on these pseudo trajectories and subsequently fine-tuned using scarce real EMA data. This approach consistently enhances AAI performance across multiple architectures, yielding significant improvements in both Pearson correlation coefficient (PCC) and root mean square error (RMSE). Trajectory analysis further confirms the physical plausibility of the generated pseudo signals, effectively alleviating the reliance on scarce EMA recordings.
📝 Abstract
Recent acoustic-to-articulatory inversion (AAI) models rely on electromagnetic articulography (EMA) data, which are costly and limited in scale. To address this limitation, we propose \textit{ArtBoost}, a novel data augmentation strategy that leverages large-scale speech--mesh datasets originally developed for speech-driven 3D facial animation to improve AAI under limited EMA supervision. \textit{ArtBoost} extracts pseudo articulatory trajectories from visible facial anchors and uses them for pre-training before fine-tuning on real EMA data. Experiments show consistent improvements in PCC and RMSE. Trajectory analyses confirm that the pseudo articulatory signals reflect physically meaningful visible articulatory dynamics. Additional evaluations across different AAI architectures demonstrate stable performance gains, indicating that \textit{ArtBoost} can be integrated into diverse AAI models. These results suggest that speech--mesh data provide an effective and scalable source of articulatory supervision for AAI. Project page: https://cau-irislab.github.io/Interspeech26-ArtBoost/
Innovation
Methods, ideas, or system contributions that make the work stand out.
Articulatory Data Augmentation
Acoustic-to-Articulatory Inversion
Speech-Mesh Dataset
Pseudo Articulatory Trajectories
Pre-training and Fine-tuning
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
