PrismaDV: Automated Task-Aware Data Unit Test Generation

2026-04-23

AI Summary

Existing automated data unit testing methods often overlook the semantic requirements imposed by downstream tasks, limiting their ability to ensure data reliability. This work proposes a task-aware data testing generation mechanism that statically analyzes task code and data characteristics to identify data access patterns and infer implicit assumptions, thereby automatically generating executable, task-oriented tests. We introduce the SIFTA framework, which integrates large language model prompting with sparse execution feedback to dynamically refine test prompts. Evaluated on a new benchmark encompassing five datasets and sixty tasks, SIFTA significantly outperforms both task-agnostic and task-specific baselines, with its automatically generated prompts surpassing those crafted manually or produced by general-purpose optimizers.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Natural Language Processing: Generation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Data is a central resource for modern enterprises, and data validation is essential for ensuring the reliability of downstream applications. However, existing automated data unit testing frameworks are largely task-agnostic: they validate datasets without considering the semantics and requirements of the code that consumes the data. We present PrismaDV, a compound AI system that analyzes downstream task code together with dataset profiles to identify data access patterns, infer implicit data assumptions, and generate task-aware executable data unit tests. To further adapt the data unit tests over time to specific datasets and downstream tasks, we propose "Selective Informative Feedback for Task Adaptation" (SIFTA), a prompt-optimization framework that leverages the scarce outcomes from the execution of data unit tests and downstream tasks. We evaluate PrismaDV on two new benchmarks spanning 60 tasks across five datasets, where it consistently outperforms both task-agnostic and task-aware baselines in generating unit tests that reflect the end-to-end impact of data errors. Furthermore, we show that with SIFTA, we can automatically learn prompts for PrismaDV's modules that outperform prompts written by hand or generated from a generic prompt optimizer. We publicly release our benchmarks and prototype implementation.

Problem

• data validation
• task-aware testing
• data unit test
• downstream tasks
• automated testing

Innovation

• task-aware data validation
• data unit test generation
• compound AI system
• prompt optimization
• SIFTA