Description and Comparative Analysis of QuRE: A New Industrial Requirements Quality Dataset

2025-08-12

AI Summary

Existing empirical studies on natural language requirement quality are hindered by scarce, small-scale, and insufficiently detailed datasets, impeding the establishment of unified evaluation criteria and collaborative research. To address this, we propose and publicly release QuRE—a large-scale, industrial-grade requirement quality dataset comprising 2,111 requirements annotated through real-world review processes and drawn from nearly a decade of industrial contracts. QuRE is distinguished by its authenticity, current largest scale, and fine-grained contextual annotations. Through descriptive statistical analyses—including lexical diversity and readability metrics—and comparative validation against both real-world and synthetic benchmarks, we demonstrate that QuRE exhibits representative linguistic characteristics and significantly richer contextual information. QuRE fills a critical gap in high-quality empirical resources for requirement quality research and establishes a new community benchmark for rigorous, reproducible investigation.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Application Domains: Humanities & Computational Social Science

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Requirements quality is central to successful software and systems engineering. Empirical research on quality defects in natural language requirements relies heavily on datasets, ideally as realistic and representative as possible. However, such datasets are often inaccessible, small, or lack sufficient detail. This paper introduces QuRE (Quality in Requirements), a new dataset comprising 2,111 industrial requirements that have been annotated through a real-world review process. Previously used for over five years as part of an industrial contract, this dataset is now being released to the research community. In this work, we furthermore provide descriptive statistics on the dataset, including measures such as lexical diversity and readability, and compare it to existing requirements datasets and synthetically generated requirements. In contrast to synthetic datasets, QuRE is linguistically similar to existing ones. However, this dataset comes with a detailed context description, and its labels have been created and used systematically and extensively in an industrial context over a period of close to a decade. Our goal is to foster transparency, comparability, and empirical rigor by supporting the development of a common gold standard for requirements quality datasets. This, in turn, will enable more sound and collaborative research efforts in the field.

Problem

• Lack of realistic industrial requirements quality datasets
• Insufficient detail in existing requirements datasets
• Need for a common gold standard in requirements quality research

Innovation

• Introduces QuRE dataset with 2,111 industrial requirements
• Annotated through real-world industrial review process
• Provides detailed context and long-term industrial validation