Exploring Data Management Challenges and Solutions in Agile Software Development: A Literature Review and Practitioner Survey

2024-02-01

AI Summary

To address core challenges in agile software development—including fragmented data integration, heterogeneous data sources, volatile data quality, and delayed responsiveness—this study integrates a systematic literature review (SLR) covering 45 studies with an empirical survey of 32 frontline practitioners, enabling the first cross-source triangulation between academic research and industrial practice. Methodologically, it combines rigorous scholarly synthesis with real-world engineering insights. The work contributes three innovations: (1) a decentralized data management paradigm tailored to agile iterations; (2) an ontology-driven, lightweight semantic modeling approach for interoperability across evolving artifacts; and (3) an automated data governance framework supporting real-time analytics. Empirical validation demonstrates significant improvements in data integration efficiency and quality, enabling faster, evidence-based decision-making. Collectively, these contributions bridge a critical gap in flexible, evolvable data governance research and practice within agile environments.

Technology Category

• Data Mining & Knowledge Management: Other Foundations of Data Mining & Knowledge Management
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Knowledge Representation and Reasoning: Ontologies

Application Category

• Semantics and Knowledge: Provenance, trust, security and privacy, and ethical issues in managing semantic data
• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Context: Managing data related to a software product and its development poses significant challenges for software projects and agile development teams. These include integrating data from diverse sources and ensuring data quality amidst continuous change and adaptation. Objective: The paper systematically explores data management challenges and potential solutions in agile projects, aiming to provide insights into data management challenges and solutions for both researchers and practitioners. Method: We employed a mixed-methods approach, including a systematic literature review (SLR) to understand the state-of-research followed by a survey with practitioners to reflect on the state-of-practice. The SLR reviewed 45 studies, identifying and categorizing data management aspects along with their associated challenges and solutions. The practitioner survey captured practical experiences and solutions from 32 industry practitioners who were significantly involved in data management to complement the findings from the SLR. Results: Our findings identified major data management challenges in practice, such as managing data integration processes, capturing diverse data, automating data collection, and meeting real-time analysis requirements. To address the challenges, solutions such as automation tools, decentralized data management practices, and ontology-based approaches have been identified. The solutions enhance data integration, improve data quality, and enable real-time decision-making by providing flexible frameworks tailored to agile project needs. Conclusion: The study pinpointed significant challenges and actionable solutions in data management for agile software development. Our findings provide practical implications for practitioners and researchers, emphasizing the development of effective data management practices and tools to address those challenges and improve project success.

Problem

• Managing data integration in agile projects
• Ensuring data quality amidst continuous change
• Automating data collection for real-time analysis

Innovation

• Automation tools enhance data integration
• Decentralized practices improve data quality
• Ontology-based approaches enable real-time decisions