RADAR: Benchmarking Language Models on Imperfect Tabular Data

2025-06-09

AI Summary

Current language models exhibit weak “data awareness” when processing imperfect tabular data—such as missing values, outliers, and logical inconsistencies—severely compromising analytical reliability. To address this gap, we introduce RADAR, the first dedicated benchmark for evaluating model robustness to real-world tabular data imperfections. Our method comprises: (1) a novel programmatic data perturbation framework covering five realistic defect types across nine domains; (2) the first incorporation of controllable table-scale variables to systematically expose model robustness bottlenecks under progressive data quality degradation; and (3) a multi-dimensional structured query set paired with a cross-scale evaluation protocol. Experiments reveal that state-of-the-art models perform well on clean tables but suffer substantial performance drops under perturbations—confirming their inadequate data perception capabilities. RADAR is publicly released, supporting extensible perturbation generation and fine-grained scale control.

Technology Category

• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Language models (LMs) are increasingly being deployed to perform autonomous data analyses. However, their data awareness -- the ability to recognize, reason over, and appropriately handle data artifacts such as missing values, outliers, and logical inconsistencies -- remains underexplored. These artifacts are especially common in real-world tabular data and, if mishandled, can significantly compromise the validity of analytical conclusions. To address this gap, we present RADAR, a benchmark for systematically evaluating data-aware reasoning on tabular data. We develop a framework to simulate data artifacts via programmatic perturbations to enable targeted evaluation of model behavior. RADAR comprises 2980 table query pairs, grounded in real-world data spanning 9 domains and 5 data artifact types. In addition to evaluating artifact handling, RADAR systematically varies table size to study how reasoning performance holds when increasing table size. Our evaluation reveals that, despite decent performance on tables without data artifacts, frontier models degrade significantly when data artifacts are introduced, exposing critical gaps in their capacity for robust, data-aware analysis. Designed to be flexible and extensible, RADAR supports diverse perturbation types and controllable table sizes, offering a valuable resource for advancing tabular reasoning.

Problem

• Evaluating LMs' ability to handle imperfect tabular data artifacts
• Assessing model robustness against missing values and inconsistencies
• Benchmarking performance degradation with increasing table sizes

Innovation

• Benchmarking LMs on imperfect tabular data
• Simulating data artifacts via programmatic perturbations
• Evaluating artifact handling across diverse domains