Beyond Transcripts: Iterative Peer-Editing with Audio Unlocks High-Quality Human Summaries of Conversational Speech

2026-05-17

AI Summary

This study addresses the scarcity of high-quality, human-annotated benchmark datasets in spoken language summarization, particularly the challenge of effectively integrating prosodic and semantic information in the absence of textual transcripts. To tackle this limitation, the authors propose an iterative peer co-editing framework that operates solely on audio inputs. Through carefully designed multimodal input representations and human annotation experiments, they construct the first benchmark for spoken summarization that jointly incorporates prosodic and lexical cues. Experimental results demonstrate that, although initial audio-based summaries contain limited informational content, their completeness significantly improves after iterative peer editing—ultimately matching the quality of both human-written text summaries and those generated by large language models. These findings validate the feasibility and effectiveness of the proposed approach under transcription-free conditions.

Abstract

There are not enough established benchmarks for the task fo speech summarization. Creating new benchmarks demands human annotation, as LLMs could embed systemic errors and bias into datasets. We test ten annotation workflows varying input modality (audio, transcript, or both) and the inclusion of editing (self or peer-editing) to investigate potential quality tradeoffs from using human annotators to summarize audio. We compare human audio-based summaries to human transcript-based summaries to track the impact of the different information modalities on summary quality. We also compare the human outputs against four LLM benchmarks (three text, one audio) to examine whether human-written summaries are less informative than highly fluent automated outputs. We find that audio-based summaries are less informative and more compressed than transcript summaries. However, iterative peer-editing with audio mitigates this difference, enabling audio-based summaries to be as informative as their transcript counterparts and LLM summaries. These findings validate iterative peer-editing among human annotators for the creation of benchmarks informed by both lexical and prosodic information. This enables crucial dataset collection even in setting where transcripts are unavailable.

Problem

• speech summarization
• human annotation
• audio-based summarization
• benchmark dataset
• summary quality

Innovation

• iterative peer-editing
• audio-based summarization
• speech summarization benchmark
• human annotation
• prosodic information