A Data Annotation Requirements Representation and Specification (DARS)

2025-12-15

AI Summary

In AI-driven connected cyber-physical systems, annotation requirements suffer from ambiguity, misalignment among stakeholders, and unverifiability. Method: This paper proposes the first annotation-requirement–oriented dual-component framework: (1) Annotation Negotiation Cards—ensuring atomicity and stakeholder alignment; and (2) Scenario-based Annotation Specifications—enabling verifiability. Integrating requirements engineering, scenario modeling, collaborative requirement elicitation, and error-attribution mapping analysis, the framework is empirically validated in an autonomous driving perception case study. Results: It covers 18 real-world annotation errors, significantly improving annotation completeness, accuracy, and consistency. This work pioneers the systematic integration of annotation requirements into the requirements engineering discipline, establishing both theoretical foundations and practical tools for trustworthy AI data governance.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agreement, Argumentation & Negotiation

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web
• Security and Privacy: Data transparency and provenance

Abstract

With the rise of AI-enabled cyber-physical systems, data annotation has become a critical yet often overlooked process in the development of these intelligent information systems. Existing work in requirements engineering (RE) has explored how requirements for AI systems and their data can be represented. However, related interviews with industry professionals show that data annotations and their related requirements introduce distinct challenges, indicating a need for annotation-specific requirement representations. We propose the Data Annotation Requirements Representation and Specification (DARS), including an Annotation Negotiation Card to align stakeholders on objectives and constraints, and a Scenario-Based Annotation Specification to express atomic and verifiable data annotation requirements. We evaluate DARS with an automotive perception case related to an ongoing project, and a mapping against 18 real-world data annotation error types. The results suggest that DARS mitigates root causes of completeness, accuracy, and consistency annotation errors. By integrating DARS into RE, this work improves the reliability of safety-critical systems using data annotations and demonstrates how engineering frameworks must evolve for data-dependent components of today's intelligent information systems.

Problem

• Addresses distinct challenges in data annotation requirements for AI systems
• Proposes DARS to represent and specify annotation-specific requirements
• Mitigates root causes of annotation errors in safety-critical systems

Innovation

• DARS framework for annotation-specific requirement representation
• Annotation Negotiation Card aligns stakeholder objectives and constraints
• Scenario-Based Specification expresses verifiable annotation requirements