🤖 AI Summary
This study addresses the challenge of achieving context-aware precision in big data quality assessment, which existing methods struggle to accomplish. The authors propose a novel approach based on knowledge graph embeddings that integrates multi-source contextual information by modeling datasets, quality dimensions, and validation rules as a knowledge graph. To enable dynamic evaluation, numeric-valued edges are introduced to weight quality metrics adaptively according to contextual relevance. Leveraging the AmpliGraph framework, the method predicts missing relationships within the graph to automatically generate tailored data quality assessment plans. Experimental validation on real-world radiation sensor data from the Lebanese Atomic Energy Commission demonstrates that the proposed technique effectively produces comprehensive and accurate data quality evaluations, confirming its feasibility and innovation.
📝 Abstract
Automated data quality assessment is crucial for managing big data, but existing solutions face challenges in achieving accurate context-aware assessment. This paper presents a novel knowledge-based approach to enhance automated data quality assessment. Our approach utilizes knowledge graph embeddings to predict missing edges between the input dataset's context representation and the relevant quality rules and dimensions within a knowledge graph representing contextual data characteristics and the required quality assessment operations. We surpass conventional practices by integrating diverse representations within the knowledge graph, drawing insights from contextual information from a thorough literature investigation. This integration allows us to develop a comprehensive and context-specific data quality assessment plan tailored to each context. Leveraging the knowledge graph improves our understanding of the input dataset's context, overcoming the limitations of traditional methods that rely solely on strict matching and overlook contextual characteristics. By injecting numerical edge attributes, we assign corresponding weights to each predicted quality measurement, providing a comprehensive data quality assessment plan for the input dataset.
To evaluate our approach, we leverage AmpliGraph, a framework developed and benchmarked by AccentureLabs. The evaluation involves employing a real-world radiation sensors dataset provided by the Lebanese Atomic Energy Commission (LAEC-CNRS). The results obtained from this evaluation demonstrate the capability of our solution to generate a comprehensive data quality assessment plan for the given input dataset.