SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks

2026-04-19

AI Summary

This work addresses the pervasive issue of training data contamination in NL2SQL benchmarks, which often leads to inflated model performance estimates. The authors propose SPENCE, a novel framework that introduces syntax-based probing by systematically generating syntactic variants of test queries and analyzing model accuracy under varying degrees of syntactic deviation, augmented with temporal context analysis. By leveraging Kendall’s tau rank correlation and bootstrap confidence intervals, SPENCE quantifies both the extent of data contamination and the temporal gradient effect across benchmarks. Empirical evaluation reveals significant contamination in established benchmarks like Spider, whereas the newer BIRD benchmark remains largely unaffected. These findings underscore the critical role of temporally aware syntactic probing in establishing reliable and trustworthy NL2SQL evaluation protocols.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflated by contamination from benchmark queries or structurally similar patterns seen during training. We introduce SPENCE (Syntactic Probing and Evaluation of NL2SQL Contamination Effects), a controlled syntactic probing framework for detecting and quantifying such contamination. SPENCE systematically generates syntactic variants of test queries for four widely used NL2SQL datasets-Spider, SParC, CoSQL, and the newer BIRD benchmark. We use SPENCE to evaluate multiple high-capacity LLMs under execution-based scoring. For each model, we measure changes in execution accuracy across increasing levels of syntactic divergence and quantify rank sensitivity using Kendall's tau with bootstrap confidence intervals. By aligning these robustness trends with benchmark release dates, we observe a clear temporal gradient: older benchmarks such as Spider exhibit the strongest negative values and thus the highest likelihood of training leakage, whereas the more recent BIRD dataset shows minimal sensitivity and appears largely uncontaminated. Together, these findings highlight the importance of temporally contextualized, syntactic-probing evaluation for trustworthy NL2SQL benchmarking.

Problem

• NL2SQL
• benchmark contamination
• syntactic probing
• training leakage
• evaluation robustness

Innovation

• syntactic probing
• benchmark contamination
• NL2SQL
• temporal evaluation
• controlled generation