LPI-RIT at LeWiDi-2025: Improving Distributional Predictions via Metadata and Loss Reweighting with DisCo

2025-08-11

AI Summary

This work addresses the pervasive annotator disagreement problem in crowdsourced labeling. We propose an enhanced perspectivist modeling approach built upon the DisCo neural architecture: (i) incorporating annotator metadata to enrich input representations; (ii) introducing a disagreement-aware loss reweighting mechanism; and (iii) jointly modeling sample-level soft label distributions and annotator-level perspective-specific evaluations. Unlike conventional methods assuming a single underlying label distribution, our framework explicitly captures the heterogeneity of annotation behaviors. Experiments on three benchmark datasets demonstrate substantial improvements in soft label prediction accuracy and perspective-aware evaluation performance. Notably, the method achieves robust gains across error distribution metrics (e.g., KL divergence, ECE) and calibration quality, validating its capacity to effectively model complex, multi-perspective labeling patterns.

Technology Category

• Machine Learning: Multi-instance/Multi-view Learning
• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The Learning With Disagreements (LeWiDi) 2025 shared task is to model annotator disagreement through soft label distribution prediction and perspectivist evaluation, modeling annotators. We adapt DisCo (Distribution from Context), a neural architecture that jointly models item-level and annotator-level label distributions, and present detailed analysis and improvements. In this paper, we extend the DisCo by incorporating annotator metadata, enhancing input representations, and modifying the loss functions to capture disagreement patterns better. Through extensive experiments, we demonstrate substantial improvements in both soft and perspectivist evaluation metrics across three datasets. We also conduct in-depth error and calibration analyses, highlighting the conditions under which improvements occur. Our findings underscore the value of disagreement-aware modeling and offer insights into how system components interact with the complexity of human-annotated data.

Problem

• Modeling annotator disagreement via soft label distribution prediction
• Enhancing DisCo with metadata and loss reweighting for better predictions
• Improving evaluation metrics through disagreement-aware modeling techniques

Innovation

• Incorporating annotator metadata into DisCo
• Enhancing input representations for better modeling
• Modifying loss functions to capture disagreement patterns