Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments

2025-09-08

AI Summary

This study addresses the challenge of subjectivity identification in value-driven argumentation, where annotator subjectivity often obscures valuable arguments. To mitigate bias arising from inter-annotator disagreement, we propose a novel direct subjectivity identification approach: a binary classification model jointly optimized with value prediction and integrated into a contrastive learning framework. Experiments demonstrate significant improvements in detecting highly subjective arguments—those most prone to annotation disagreement—while reducing reliance on fine-grained, label-level subjectivity annotations. Further analysis uncovers latent value patterns underlying annotation discrepancies, revealing systematic relationships between argument content, value orientations, and subjective interpretation. These findings provide both theoretical insight and empirical support for developing more robust, interpretable, and value-aware argument analysis systems. (149 words)

Technology Category

• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Knowledge Representation and Reasoning: Argumentation
• Multiagent Systems: Agreement, Argumentation & Negotiation

Application Category

• Web Mining and Content Analysis: Sentiment analysis and opinion mining
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process.

Problem

• Identifying subjectivity in value-laden arguments
• Methods for recognizing human values motivating arguments
• Distinguishing subjective arguments requiring nuanced interpretation

Innovation

• Direct subjectivity identification improves model performance
• Combining contrastive and cross-entropy loss reduces label dependency
• Methods identify differently interpreted arguments for nuanced annotation