MAGPIE: Multi-Task Analysis of Media-Bias Generalization with Pre-Trained Identification of Expressions

2024-02-27

International Conference on Language Resources and Evaluation

AI Summary

Media bias detection suffers from poor generalization and overreliance on small-scale, single-task datasets. To address this, we propose MAGPIE, a multi-task pre-training framework built upon LBM—the first large-scale, multi-task benchmark for media bias comprising 59 diverse tasks—unifying bias identification with related tasks such as sentiment and emotion analysis. Our key contributions include: (i) introducing the first multi-task pre-training paradigm specifically for media bias; (ii) constructing LBM, the largest and most comprehensive bias-related multi-task dataset to date; (iii) empirically demonstrating positive transfer from sentiment and emotion tasks to bias detection; and (iv) showing that scaling task count significantly improves performance. MAGPIE employs a RoBERTa backbone with a shared encoder and task-decoupled heads, augmented by task interference analysis and collaborative learning strategies. On BABE, it achieves a +3.3% F1 gain, reduces fine-tuning steps to 15% of single-task baselines, and substantially enhances cross-task generalization and training efficiency.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Media bias detection poses a complex, multifaceted problem traditionally tackled using single-task models and small in-domain datasets, consequently lacking generalizability. To address this, we introduce MAGPIE, a large-scale multi-task pre-training approach explicitly tailored for media bias detection. To enable large-scale pre-training, we construct Large Bias Mixture (LBM), a compilation of 59 bias-related tasks. MAGPIE outperforms previous approaches in media bias detection on the Bias Annotation By Experts (BABE) dataset, with a relative improvement of 3.3% F1-score. Furthermore, using a RoBERTa encoder, we show that MAGPIE needs only 15% of fine-tuning steps compared to single-task approaches. We provide insight into task learning interference and show that sentiment analysis and emotion detection help learning of all other tasks, and scaling the number of tasks leads to the best results. MAGPIE confirms that MTL is a promising approach for addressing media bias detection, enhancing the accuracy and efficiency of existing models. Furthermore, LBM is the first available resource collection focused on media bias MTL.

Problem

• Detecting media bias lacks generalizability with single-task models
• Large-scale multi-task pre-training improves media bias detection accuracy
• Compiling diverse bias-related tasks enhances model performance and efficiency

Innovation

• Multi-task pre-training for media bias detection
• Large Bias Mixture (LBM) with 59 tasks
• RoBERTa encoder reduces finetuning steps significantly