Can large audio language models understand child stuttering speech? speech summarization, and source separation

2025-10-21

AI Summary

This study presents the first systematic evaluation of large audio-language models (LALMs) for stuttered child speech understanding, focusing on speech summarization and single-channel speaker separation. Addressing key challenges—including high acoustic variability in children’s voices, prominent disfluencies (repetitions, prolongations, blocks), and bias from adult speech priors—we design a clinically informed summarization objective that preserves stuttering-relevant features and develop a child-specific evaluation framework. Our methodology integrates LALMs with self-supervised source separation, LLM-based automated assessment, expert human evaluation, and BERTScore (F1) for multi-dimensional validation. Key contributions include: (1) characterizing the performance limits and failure modes of LALMs on stuttered child speech; (2) releasing reproducible prompt templates and evaluation scripts; and (3) establishing a technical foundation for clinical screening and educational support applications.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Child speech differs from adult speech in acoustics, prosody, and language development, and disfluencies (repetitions, prolongations, blocks) further challenge Automatic Speech Recognition (ASR) and downstream Natural Language Processing (NLP). Recent large audio-language models (LALMs) demonstrate strong cross-modal audio understanding; however, their behavior in disfluent child speech remains underexplored. We evaluate several state-of-the-art LALMs in two settings: an interview (mixed speakers) and a reading task (single child). The tasks are (i) single-channel source separation to isolate the child and (ii) child-only summarization that preserves clinically relevant disfluencies and avoids adult-speech leakage. Evaluation combines Large Language Model (LLM) as a judge, human expert ratings, and BERTScore (F1), and we report agreement between models and between models and humans to assess reliability. Our findings delineate the conditions under which LALMs produce faithful child-only summaries from mixed audio and where they fail, offering practical guidance for clinical and educational deployments. We provide prompts and evaluation scripts to support replication.

Problem

• Evaluating LALMs' ability to understand disfluent child speech
• Assessing child speech source separation and disfluency-preserving summarization
• Testing model reliability through human and automated evaluation methods

Innovation

• Large audio-language models for child stuttering speech
• Source separation to isolate child speech from mixed audio
• Summarization preserving clinically relevant disfluencies