Multi-Perspective Stance Detection

2024-11-13

HHAI Workshops

AI Summary

Traditional stance detection forces crowd-sourced annotations into a single consensus label, ignoring legitimate inter-annotator disagreement arising from diverse backgrounds—thereby compromising model robustness and fairness. This work pioneers modeling stance annotation as a multi-perspective classification task, explicitly capturing annotator viewpoint diversity and its influence on prediction confidence. We propose a multi-perspective supervised learning framework that jointly models the crowd annotation distribution and incorporates uncertainty-aware inference, enhancing ethical compatibility without sacrificing predictive performance. Experiments across multiple stance detection benchmarks demonstrate substantial improvements over single-label baselines. Our approach strengthens robustness against subjectivity-induced biases, improves decision interpretability, and advances the co-evolution of responsible AI and technical performance.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Subjective NLP tasks usually rely on human annotations provided by multiple annotators, whose judgments may vary due to their diverse backgrounds and life experiences. Traditional methods often aggregate multiple annotations into a single ground truth, disregarding the diversity in perspectives that arises from annotator disagreement. In this preliminary study, we examine the effect of including multiple annotations on model accuracy in classification. Our methodology investigates the performance of perspective-aware classification models in stance detection task and further inspects if annotator disagreement affects the model confidence. The results show that multi-perspective approach yields better classification performance outperforming the baseline which uses the single label. This entails that designing more inclusive perspective-aware AI models is not only an essential first step in implementing responsible and ethical AI, but it can also achieve superior results than using the traditional approaches.

Problem

• Investigating multi-annotator perspectives in stance detection NLP tasks
• Evaluating perspective-aware models versus single-label baseline approaches
• Examining how annotator disagreement affects model confidence and performance

Innovation

• Uses perspective-aware classification models for stance detection
• Incorporates multiple annotations instead of single ground truth
• Analyzes annotator disagreement impact on model confidence