Embracing Diversity: A Multi-Perspective Approach with Soft Labels

2025-03-01

AI Summary

To address subjectivity and label noise in stance detection arising from annotator background discrepancies, this paper proposes a perspective-aware multi-source soft-labeling framework. Methodologically, we introduce the first dual-source collaborative annotation scheme integrating human experts and large language models, explicitly modeling annotation diversity and stance subjectivity, and design an uncertainty-aware multi-perspective ensemble classifier. Our contributions are threefold: (1) We release the first stance detection dataset featuring both human and AI annotations alongside fine-grained perspective metadata; (2) Our soft-label learning approach achieves significant F1-score improvements across multiple benchmarks; (3) The model produces better-calibrated confidence scores, jointly optimizing classification accuracy and decision accountability. Experiments demonstrate that explicit diversity modeling enhances both robustness and ethical compliance.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Calibration & Uncertainty Quantification
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Prior studies show that adopting the annotation diversity shaped by different backgrounds and life experiences and incorporating them into the model learning, i.e. multi-perspective approach, contribute to the development of more responsible models. Thus, in this paper we propose a new framework for designing and further evaluating perspective-aware models on stance detection task,in which multiple annotators assign stances based on a controversial topic. We also share a new dataset established through obtaining both human and LLM annotations. Results show that the multi-perspective approach yields better classification performance (higher F1-scores), outperforming the traditional approaches that use a single ground-truth, while displaying lower model confidence scores, probably due to the high level of subjectivity of the stance detection task.

Problem

• Develops a framework for perspective-aware stance detection models.
• Incorporates diverse annotations from multiple human and LLM sources.
• Improves classification performance over traditional single-ground-truth methods.

Innovation

• Multi-perspective approach with soft labels
• New framework for perspective-aware models
• Dataset combining human and LLM annotations