Summarization of Opinionated Political Documents with Varied Perspectives

2024-11-06

International Conference on Computational Linguistics

AI Summary

To address escalating political polarization, this paper introduces the novel task of “Multi-Perspective Independent Summarization”: generating faithful, stance-specific summaries (e.g., left/center/right) from a single opinionated news article to mitigate cognitive bias. We construct the first fine-grained, multi-stance news summarization dataset with human-annotated stance labels and summary pairs. Furthermore, we propose a comprehensive evaluation framework jointly measuring stance fidelity, information coverage, and linguistic quality. Using this framework, we systematically benchmark ten state-of-the-art large language models—including GPT-4o—on the task. Results reveal pervasive stance distortion across all models, particularly in stance transfer and preservation of key claims; even GPT-4o, the top-performing model, falls significantly short of acceptable fidelity. This work is the first to empirically expose systematic failures of LLMs in political perspective modeling, establishing both theoretical foundations and an empirical benchmark for trustworthy, interpretable political content summarization.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Global partisan hostility and polarization has increased, and this polarization is heightened around presidential elections. Models capable of generating accurate summaries of diverse perspectives can help reduce such polarization by exposing users to alternative perspectives. In this work, we introduce a novel dataset and task for independently summarizing each political perspective in a set of passages from opinionated news articles. For this task, we propose a framework for evaluating different dimensions of perspective summary performance. We benchmark 10 models of varying sizes and architectures through both automatic and human evaluation. While recent models like GPT-4o perform well on this task, we find that all models struggle to generate summaries faithful to the intended perspective. Our analysis of summaries focuses on how extraction behavior depends on the features of the input documents.

Problem

• Summarizing diverse political perspectives in opinionated documents
• Evaluating model performance on perspective-specific summarization tasks
• Assessing faithfulness of summaries to original political viewpoints

Innovation

• Novel dataset for summarizing political perspectives
• Framework evaluating perspective summary performance
• Benchmarking 11 models via automatic and human evaluation