Hotter and Colder: A New Approach to Annotating Sentiment, Emotions, and Bias in Icelandic Blog Comments

2025-02-24

AI Summary

This work addresses the critical lack of high-quality annotated resources for detecting harmful online behaviors—including sentiment, emotion, bias, hate speech, and group generalizations—in Icelandic-language online reviews. We introduce the first large-scale, multitask Icelandic blog comment dataset, comprising 12,232 unique comments and 19,301 high-quality annotations across 25 fine-grained behavioral categories. Our methodology employs a two-stage paradigm: (1) high-confidence automated pre-screening using GPT-4o mini, integrated with probability-driven sampling and quantification via a 5-point Likert scale; and (2) rigorous human verification and refinement through crowdsourcing. This approach enables the first reproducible, multidimensional, and fine-grained analysis of content safety in Icelandic. It substantially improves benchmark performance across related tasks and fills a longstanding gap in both data and methodology for harmful content detection in Nordic low-resource languages.

Technology Category

• Natural Language Processing: Safety and Robustness
• Humans and AI: Crowd Sourcing and Human Computation
• Application Domains: Humanities & Computational Social Science

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Large pretrained models with web data
• Responsible Web: Mitigating online harms, including harassment, spam, and fake reviews

Abstract

This paper presents Hotter and Colder, a dataset designed to analyze various types of online behavior in Icelandic blog comments. Building on previous work, we used GPT-4o mini to annotate approximately 800,000 comments for 25 tasks, including sentiment analysis, emotion detection, hate speech, and group generalizations. Each comment was automatically labeled on a 5-point Likert scale. In a second annotation stage, comments with high or low probabilities of containing each examined behavior were subjected to manual revision. By leveraging crowdworkers to refine these automatically labeled comments, we ensure the quality and accuracy of our dataset resulting in 12,232 uniquely annotated comments and 19,301 annotations. Hotter and Colder provides an essential resource for advancing research in content moderation and automatically detectiong harmful online behaviors in Icelandic.

Problem

• Annotating Icelandic blog comments
• Detecting harmful online behaviors
• Advancing content moderation research

Innovation

• GPT-4o mini annotation
• 5-point Likert scale
• crowdworker manual revision