A German Gold-Standard Dataset for Sentiment Analysis in Software Engineering

2025-07-09

AI Summary

Existing German sentiment analysis for software engineering is hindered by the scarcity of high-quality, domain-adapted annotated data. To address this, we introduce GERSEMO—the first gold-standard German sentiment dataset specifically designed for software engineering, curated from 5,949 utterances sourced from the Android-Hilfe.de forum. Annotations follow Shaver’s six-dimensional basic emotion model and were performed independently by four native German-speaking computer science students under double-blind conditions, achieving high inter-annotator agreement (Krippendorff’s α = 0.82). GERSEMO fills a critical gap in non-English, domain-specific sentiment resources. Empirical evaluation reveals that state-of-the-art German sentiment tools suffer substantial performance degradation in software engineering contexts—exhibiting an average 37.6% drop in F1 score—demonstrating GERSEMO’s essential role in enabling accurate, domain-informed sentiment recognition.

Technology Category

• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Application Domains: Software Engineering
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Web Mining and Content Analysis: Sentiment analysis and opinion mining
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Sentiment analysis is an essential technique for investigating the emotional climate within developer teams, contributing to both team productivity and project success. Existing sentiment analysis tools in software engineering primarily rely on English or non-German gold-standard datasets. To address this gap, our work introduces a German dataset of 5,949 unique developer statements, extracted from the German developer forum Android-Hilfe.de. Each statement was annotated with one of six basic emotions, based on the emotion model by Shaver et al., by four German-speaking computer science students. Evaluation of the annotation process showed high interrater agreement and reliability. These results indicate that the dataset is sufficiently valid and robust to support sentiment analysis in the German-speaking software engineering community. Evaluation with existing German sentiment analysis tools confirms the lack of domain-specific solutions for software engineering. We also discuss approaches to optimize annotation and present further use cases for the dataset.

Problem

• Lack of German gold-standard datasets for sentiment analysis in software engineering
• Need for domain-specific sentiment analysis tools in German software engineering
• Validation and robustness of a new German dataset for developer sentiment

Innovation

• German dataset for sentiment analysis
• Annotated with six basic emotions
• High interrater agreement and reliability