Transformer-based segmentation of prosodic boundaries in Brazilian Portuguese

2026-07-08

AI Summary

This study addresses the lack of effective deep learning approaches for automatic prosodic boundary detection in Brazilian Portuguese by proposing the first application of the Whisper large-v3 model to this task. Through fine-tuning, the model enables end-to-end simultaneous prediction of speech transcription and prosodic boundary labels. The approach integrates multimodal cues and incorporates a test-time filtering strategy alongside n-gram and acoustic visualization analyses to systematically evaluate the model’s sensitivity to syntactic, semantic, and prosodic features. Evaluated on the NURC-SP test set, the method achieves an F1 score of 0.731, and demonstrates strong generalization with an F1 score of 0.796 on the out-of-domain MuPe-Diversidades dataset, significantly outperforming existing approaches.

Abstract

Automatic prosodic segmentation identifies boundaries between speech units from acoustic and linguistic evidence. Although recent deep learning approaches have produced strong results for English, automatic segmentation for Brazilian Portuguese (BP) still relies mostly on rule-based or traditional machine-learning methods. This paper presents SAMPA, a Whisper-based segmenter that transcribes BP speech while inserting explicit markers for terminal prosodic boundaries. We fine-tune Whisper large-v3 on manually segmented recordings from the NURC-SP dataset and evaluate different training and test-time filtering configurations, including out-of-distribution testing on the MuPe-Diversidades dataset. SAMPA achieves competitive boundary-detection performance across settings, with the best models reaching F1=0.731 on the held-out test split and F1=0.796 on MuPe-Diversidades. Finally, through n-gram and acoustic-visual analyses, we show that our model follows morphosyntactic, semantic, and prosodic cues for detecting prosodic boundaries.

Problem

• prosodic segmentation
• Brazilian Portuguese
• prosodic boundaries
• automatic speech processing
• speech units

Innovation

• Transformer
• prosodic segmentation
• Whisper
• Brazilian Portuguese
• fine-tuning