Codebook-Injected Dialogue Segmentation for Multi-Utterance Constructs Annotation: LLM-Assisted and Gold-Label-Free Evaluation

2026-01-17

AI Summary

This study addresses the limitations of traditional dialogue act (DA) annotation, which is confined to single utterances and often leads to inconsistent boundary delineation and reduced reliability. The authors propose a codebook-injected dialogue segmentation approach that integrates large language models (LLMs) with downstream DA annotation guidelines to optimize multi-utterance segmentation without gold-standard labels. They introduce, for the first time, an unsupervised evaluation framework that assesses segmentation quality through metrics such as segment consistency, discriminability, and human–machine distribution alignment. Experimental results demonstrate that DA-aware segmentation achieves higher internal consistency than text-only baselines; LLMs excel at generating semantically coherent segments, while coherence-based baselines better capture global discourse shifts, suggesting that segmentation strategies should be tailored to specific task objectives.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Segmentation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Dialogue Act (DA) annotation typically treats communicative or pedagogical intent as localized to individual utterances or turns. This leads annotators to agree on the underlying action while disagreeing on segment boundaries, reducing apparent reliability. We propose codebook-injected segmentation, which conditions boundary decisions on downstream annotation criteria, and evaluate LLM-based segmenters against standard and retrieval-augmented baselines. To assess these without gold labels, we introduce evaluation metrics for span consistency, distinctiveness, and human-AI distributional agreement. We found DA-awareness produces segments that are internally more consistent than text-only baselines. While LLMs excel at creating construct-consistent spans, coherence-based baselines remain superior at detecting global shifts in dialogue flow. Across two datasets, no single segmenter dominates. Improvements in within-segment coherence frequently trade off against boundary distinctiveness and human-AI distributional agreement. These results highlight segmentation as a consequential design choice that should be optimized for downstream objectives rather than a single performance score.

Problem

• Dialogue Segmentation
• Dialogue Act Annotation
• Boundary Disagreement
• Annotation Reliability
• Multi-Utterance Constructs

Innovation

• codebook-injected segmentation
• dialogue act-aware segmentation
• gold-label-free evaluation
• span consistency
• LLM-assisted dialogue analysis