Data Quality in Crowdsourcing and Spamming Behavior Detection

2024-04-04

arXiv.org

AI Summary

Evaluating annotation quality in crowdsourcing without ground-truth labels remains challenging. Method: This paper proposes the first unified evaluation framework based on variance decomposition, jointly modeling annotation consistency and annotator reliability. It introduces a novel spammer index and two reliability metrics, and designs a three-tier cheating-behavior-pattern-driven detection mechanism integrating variance decomposition, generalized random-effects modeling, and Markov chain modeling. Contribution/Results: Evaluated on synthetic data and real-world face verification tasks from two crowdsourcing platforms, the method significantly improves annotation quality estimation accuracy, effectively identifies and filters low-quality annotations, and enhances the robustness of downstream machine learning models.

Technology Category

• Humans and AI: Crowd Sourcing and Human Computation
• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

As crowdsourcing emerges as an efficient and cost-effective method for obtaining labels for machine learning datasets, it is important to assess the quality of crowd-provided data, so as to improve analysis performance and reduce biases in subsequent machine learning tasks. Given the lack of ground truth in most cases of crowdsourcing, we refer to data quality as annotators' consistency and credibility. Unlike the simple scenarios where Kappa coefficient and intraclass correlation coefficient usually can apply, online crowdsourcing requires dealing with more complex situations. We introduce a systematic method for evaluating data quality and detecting spamming threats via variance decomposition, and we classify spammers into three categories based on their different behavioral patterns. A spammer index is proposed to assess entire data consistency and two metrics are developed to measure crowd worker's credibility by utilizing the Markov chain and generalized random effects models. Furthermore, we showcase the practicality of our techniques and their advantages by applying them on a face verification task with both simulation and real-world data collected from two crowdsourcing platforms.

Problem

• Assessing quality of crowd-provided data for machine learning
• Detecting spamming behavior in online crowdsourcing tasks
• Evaluating annotators' consistency and credibility without ground truth

Innovation

• Variance decomposition for data quality evaluation
• Spammer index for data consistency assessment
• Markov chain and random effects for credibility metrics