🤖 AI Summary
This work addresses the challenge of schema drift in data pipelines, which often manifests at runtime, causing errors and increasing maintenance overhead. The authors propose a lightweight framework built on Scala 3 that enforces structural contracts between producers and consumers at compile time. By leveraging the type system and compile-time metaprogramming, the framework automatically derives Spark DataFrame schemas from shared contracts and validates actual data structures prior to ingestion. It combines compile-time guarantees with policy-aware runtime comparators to support nested and optional fields as well as subset semantics, thereby ensuring both forward and backward compatibility. Empirical evaluation demonstrates the framework’s effectiveness in end-to-end workflows, with reproducible benchmarking conducted across two distinct environments.
📝 Abstract
Schema drift in data pipelines is often caught only when a job touches real data. Typed-Dataset layers close part of this gap but require wholesale adoption; table-level enforcement systems close another part but operate at write time against a stored schema. We present a small Scala 3 framework that occupies the seam: it proves producer-to-contract structural compatibility under explicit policies at compile time, derives Spark schemas from the same contract types, and re-checks the actual DataFrame schema at the sink boundary before write. The artifact fuses the compile-time witness with a policy-aware runtime comparator that adds a nested-collection-optionality check Spark's built-in comparators omit and implements structural subset semantics for backward- and forward-compatible field sets. Evaluation covers compile-time proofs, runtime policy tests, builder-path end-to-end tests, and reproducible benchmarks on two environments. This is a narrow, honest mechanism artifact; the broader claim that compile-time structural contracts deliver measurable productivity or reliability in practice is stated as motivation and left for future work.