Improving Fairness of Large Language Models in Multi-document Summarization

2025-06-09

AI Summary

To address fairness deficits in multi-document summarization (MDS) arising from sociodemographic biases, this paper introduces FairPO—a preference optimization framework that unifies modeling of both summary-level and corpus-level fairness, the first such approach in MDS. Methodologically, FairPO constructs fairness-aware preference pairs by perturbing document sets along sociodemographic attributes and employs a dynamically weighted loss to jointly optimize fairness objectives at both levels. Integrating preference learning with large language model (LLM) fine-tuning, FairPO achieves substantial improvements in corpus-level fairness (+12.7% across multiple MDS benchmarks) while preserving ROUGE-L scores and factual consistency. This work establishes a scalable, preference-driven paradigm for fairness-aware LLM-based summarization.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Fairness in multi-document summarization (MDS) is crucial for providing comprehensive views across documents with diverse social attribute values, which can significantly impact decision-making. For example, a summarization system that tends to overrepresent negative reviews of products can mislead customers into disregarding good products. Previous works measure fairness in MDS at two levels: summary-level and corpus-level. While summary-level fairness focuses on individual summaries, corpus-level fairness focuses on a corpus of summaries. Recent methods primarily focus on summary-level fairness. We propose FairPO, a preference tuning method that focuses on both summary-level and corpus-level fairness in MDS. To improve summary-level fairness, we propose to generate preference pairs by perturbing document sets. To improve corpus-level fairness, we propose fairness-aware preference tuning by dynamically adjusting the weights of preference pairs. Our experiments show that FairPO outperforms strong baselines while maintaining the critical qualities of summaries. The code is available at https://github.com/leehaoyuan/coverage_fairnes.

Problem

• Ensuring fairness in multi-document summarization across diverse social attributes
• Balancing summary-level and corpus-level fairness in generated summaries
• Improving fairness without compromising critical summary qualities

Innovation

• Preference tuning method for multi-document summarization
• Generates preference pairs by perturbing document sets
• Dynamically adjusts weights for fairness-aware tuning