"Will This Data Break My Task?" - Interactive Synthesis of Task-Aware Data Unit Tests

2026-08-10

AI Summary

Existing data unit testing approaches predominantly rely on observed data generation and largely overlook the semantics of downstream task code, limiting their ability to ensure data suitability for specific tasks. This work proposes PrismaDV, a novel system that integrates task code semantics into the test generation process for the first time. Leveraging large language models, PrismaDV jointly analyzes tabular data and associated task code through a pipeline comprising data profiling, column access detection, data flow tracing, and implicit assumption inference to automatically generate task-aware data unit tests. It further constructs an interactive data-code assumption graph to trace the provenance of inferred constraints. Experimental evaluation across five real-world datasets and 60 downstream tasks demonstrates that PrismaDV significantly outperforms task-agnostic baselines and enables users to interactively refine test logic via a visual interface.

Abstract

Data is a central resource for modern enterprises and institutions, and data errors propagating through data pipelines lead to serious impact in production. Therefore, data validation is essential for ensuring the reliability of downstream applications. This led to the development of data unit tests, executable programs that test data before moving it around through large data pipelines. However, existing frameworks derive data unit tests from observed data alone, ignoring the semantics of the code that consumes the data downstream. To this end, we present PrismaDV, a compound AI system that synthesizes task-aware data unit tests for tabular data by jointly analyzing data and downstream task code. PrismaDV decomposes the test generation into multiple LLM-powered steps: data profiling, detection of column accesses, data flow analysis in the task code, and the inference of implicit data assumptions. It subsequently synthesizes code for the data unit test, and maintains an internal ``data-code assumption graph'' that links generated data constraints back to the task's source code. We demonstrate PrismaDV through an interactive web-based interface where attendees run the system on five real-world datasets with 60 downstream tasks, synthesize, inspect and refine both natural language assumptions about the data and executable data constraints. The interface allows attendees to navigate the data-code assumption graph, compare task-aware data unit tests against task-agnostic baselines on erroneous data batches, and interactively edit assumptions and data constraints. Furthermore, attendees can observe how a custom prompt optimizer adapts the system to specific datasets over time.

Problem

• data validation
• data unit tests
• task-aware
• downstream tasks
• data errors

Innovation

• task-aware data validation
• data unit tests
• compound AI system
• data-code assumption graph
• LLM-powered test synthesis