Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models

2025-09-24

AI Summary

Current Transformer-based language models inadequately model backchannels and fillers—critical discourse markers in dialogue—thereby limiting conversational naturalness. This paper proposes three fine-tuning strategies tailored to multilingual (English/Japanese) conversational corpora to systematically enhance models’ discrimination and generation capabilities for these functionally distinct utterance types. We rigorously evaluate improvements via clustering analysis, silhouette coefficient assessment, and standard NLG metrics. Results show that fine-tuned models exhibit significantly improved structural clarity in representation space (higher silhouette scores) and generate utterances more closely aligned with human-produced speech. Our key contribution is the first integration of fine-grained backchannel–filler distinction into language model fine-tuning objectives, establishing a reproducible methodology and empirical foundation for developing more human-like conversational systems.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Backchannels and fillers are important linguistic expressions in dialogue, but are under-represented in modern transformer-based language models (LMs). Our work studies the representation of them in language models using three fine-tuning strategies. The models are trained on three dialogue corpora in English and Japanese, where backchannels and fillers are preserved and annotated, to investigate how fine-tuning can help LMs learn their representations. We first apply clustering analysis to the learnt representation of backchannels and fillers, and have found increased silhouette scores in representations from fine-tuned models, which suggests that fine-tuning enables LMs to distinguish the nuanced semantic variation in different backchannel and filler use. We also use natural language generation (NLG) metrics to confirm that the utterances generated by fine-tuned language models resemble human-produced utterances more closely. Our findings suggest the potentials of transforming general LMs into conversational LMs that are more capable of producing human-like languages adequately.

Problem

• Under-representation of backchannels and fillers in transformer language models
• Improving language models' ability to distinguish nuanced semantic variations
• Transforming general LMs into conversational models producing human-like language

Innovation

• Fine-tuning language models on dialogue corpora
• Clustering analysis to distinguish semantic variations
• NLG metrics validate human-like utterance generation