A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization

2026-06-14

AI Summary

This work addresses critical limitations in current dialogue summarization evaluation—namely, narrow scenario coverage, constrained input lengths, insufficient sample sizes, and a lack of fine-grained assessment of state-of-the-art reasoning systems and efficient small models. To this end, the authors introduce OmniCSEval, a unified benchmark comprising 1,800 long-context dialogues (ranging from 128 to 32k tokens) across six real-world scenarios. They further develop a bidirectional fact-checking framework that combines human-in-the-loop key fact extraction with multi-LLM consensus-based fact decomposition for reliable, multidimensional evaluation. Empirical evaluation of 28 large language models reveals substantial cross-scenario summarization challenges, elucidating the nuanced interplay between model scale and reasoning capability, thereby offering practical guidance for model selection in real-world deployment.

Abstract

Despite the significant advancement of LLMs in conversation summarization, their evaluation remains limited by insufficient scenarios, input lengths, and sample sizes. Furthermore, existing benchmarks often omit frontier reasoning systems and efficient small models, or lack fine-grained, multi-dimensional assessments. To bridge these gaps, we propose OmniCSEval, a unified benchmark comprising 1,800 diverse conversations across six real-world scenarios, featuring context lengths ranging from 128 to 32k tokens. For fine-grained evaluation, we employ a bidirectional fact-checking framework that integrates key fact matching to assess completeness and conciseness, alongside summary fact verification to evaluate faithfulness. To ensure reliable assessment, we establish a human-LLM collaborative pipeline for key fact extraction and a multi-LLM consensus verifier for summary fact decomposition. Leveraging this framework, we evaluate 28 LLMs across four distinct categories grouped by reasoning capability and model scale. Our extensive empirical study reveals critical insights regarding the cross-scenario challenges current LLMs continue to face, the impacts of reasoning and scale, and the efficiency and adaptability of reasoning models. We also provide guidance for system selection in real-world deployments.

Problem

• conversation summarization
• large language models
• evaluation benchmark
• multi-dimensional assessment
• model scalability

Innovation

• conversation summarization
• large language models
• multi-dimensional evaluation
• fact-checking framework
• OmniCSEval