Evaluating Discourse Cohesion in Pre-trained Language Models

2025-03-08

CODI

AI Summary

This work evaluates pretrained language models’ capacity for discourse cohesion. Addressing the lack of systematic assessment in prior research, we introduce *discourse_cohesion*, the first fine-grained benchmark covering both adjacent-sentence and cross-sentence cohesion phenomena—including coreference, connectives, ellipsis, and substitution—built from manually curated, controlled examples. We evaluate BERT, RoBERTa, GPT, and others via zero-shot and few-shot prompting, performing consistency scoring and error analysis. Key contributions include: (1) the first explicit definition and quantification of discourse cohesion ability; (2) a multi-phenomenon, cross-sentence evaluation framework; and (3) an open-source, reproducible benchmark. Experiments reveal significant performance degradation on non-adjacent cohesion tasks (e.g., paragraph-level coreference resolution) and a strong correlation between model performance and pretraining objectives (MLM vs. autoregressive), providing critical empirical guidance for architectural and objective design improvements.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Large pre-trained neural models have achieved remarkable success in natural language process (NLP), inspiring a growing body of research analyzing their ability from different aspects. In this paper, we propose a test suite to evaluate the cohesive ability of pre-trained language models. The test suite contains multiple cohesion phenomena between adjacent and non-adjacent sentences. We try to compare different pre-trained language models on these phenomena and analyze the experimental results,hoping more attention can be given to discourse cohesion in the future. The built discourse cohesion test suite will be publicly available at https://github.com/probe2/discourse_cohesion.

Problem

• Evaluate cohesive ability of pre-trained language models.
• Compare models on multiple cohesion phenomena.
• Encourage future research on discourse cohesion.

Innovation

• Test suite evaluates pre-trained model cohesion
• Analyzes cohesion in adjacent and non-adjacent sentences
• Compares cohesion abilities across different language models