GitReq: A Gold Standard Dataset for Software Quality Requirements

2026-06-19

AI Summary

This work addresses the scarcity of fine-grained, expert-annotated non-functional requirement (NFR) samples in existing public GitHub datasets by introducing GitReq—the first large-scale GitHub-based quality requirements dataset aligned with the ISO/IEC 25010 standard. GitReq comprises 6,302 expert-validated requirements extracted from 4,080 repositories, spanning all eight quality characteristics defined in the standard. The construction methodology employs category-specific tri-signal mining, a preprocessing step to separate functional from non-functional requirements, and a rigorous manual annotation protocol, achieving a Fleiss’ Kappa inter-annotator agreement of 0.72. Experimental evaluation demonstrates that GPT-5.2 attains a macro-averaged F1 score of 0.641 under zero-shot settings, confirming both the dataset’s validity and its inherent challenge for current language models.

Abstract

GitHub issue trackers contain millions of developer-written quality concerns, including performance bottlenecks and security vulnerabilities, yet no publicly available GitHub dataset classifies these into fine-grained software quality categories. We construct and release GitReq GitHub Requirement Issue, comprising 6,302 expert-validated requirements mined from 55,588 raw GitHub candidates across 4,080 repositories, labeled across eight ISO/IEC 25010:2011-aligned categories: Performance, Security, Portability, Availability, Fault-tolerance, Scalability, Maintainability, and a Functional baseline. Dataset construction involved category-specific triple-signal GitHub mining, separate non-functional requirement (NFR) and functional requirement (FR) preprocessing pipelines with per-category parameters, and expert human annotation achieving substantial inter-annotator agreement (Fleiss' Kappa~=~0.72). Zero-shot evaluation with four large language models (LLMs) establishes baselines, with GPT-5.2 reaching the highest macro-averaged F1 of 0.641. GitReq is publicly released with full materials to advance research in automated requirement classification and software quality analysis.

Problem

• software quality requirements
• GitHub issue classification
• non-functional requirements
• requirement categorization
• ISO/IEC 25010

Innovation

• Gold Standard Dataset
• Software Quality Requirements
• GitHub Issue Mining
• Non-Functional Requirements Classification
• LLM Zero-shot Evaluation