Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertion

2024-12-17

arXiv.org

AI Summary

Large language models (LLMs) generate speech lacking human-like disfluencies—such as fillers, repetitions, and self-corrections—resulting in low perceptual spontaneity and limited anthropomorphism in conversational AI agents. Method: This paper proposes a controllable disfluency modeling framework: the first to formulate discrete spoken disfluencies as a learnable, post-hoc LLM enhancement module; integrates LoRA-efficient fine-tuning with end-to-end text-to-speech (TTS) optimization supporting prosodic and paralinguistic features; and enables fine-grained control via disfluency-pattern injection through prompt engineering. Contribution/Results: User studies demonstrate significant improvements in perceived spontaneity and overall speech naturalness—achieving state-of-the-art performance—while incurring only marginal intelligibility degradation. The approach establishes a novel paradigm for humanizing conversational AI speech through structured, controllable disfluency synthesis.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Disfluencies are a natural feature of spontaneous human speech but are typically absent from the outputs of Large Language Models (LLMs). This absence can diminish the perceived naturalness of synthesized speech, which is an important criteria when building conversational agents that aim to mimick human behaviours. We show how the insertion of disfluencies can alleviate this shortcoming. The proposed approach involves (1) fine-tuning an LLM with Low-Rank Adaptation (LoRA) to incorporate various types of disfluencies into LLM-generated utterances and (2) synthesizing those utterances using a text-to-speech model that supports the generation of speech phenomena such as disfluencies. We evaluated the quality of the generated speech across two metrics: intelligibility and perceived spontaneity. We demonstrate through a user study that the insertion of disfluencies significantly increase the perceived spontaneity of the generated speech. This increase came, however, along with a slight reduction in intelligibility.

Problem

• LLM-generated speech lacks natural disfluencies found in human speech
• Absence of disfluencies reduces perceived naturalness in conversational agents
• Proposing disfluency insertion method to enhance speech spontaneity

Innovation

• Fine-tuning LLM with LoRA for disfluency insertion
• Synthesizing speech with TTS supporting disfluency generation
• Evaluating speech quality via intelligibility and perceived spontaneity