Database Context Compression for Text-to-SQL on Real-World Large Databases

2026-06-26

AI Summary

This work addresses the limited performance of existing Text-to-SQL methods on real-world large-scale databases, which stems primarily from redundant and inefficient database context representations. To overcome this, the authors propose the SGCF principle—Structural Deduplication, Homogeneous Templating, Semantic Componentization, and Evidence Purification—and formulate context compression as a query-agnostic preprocessing step. They introduce DBCC, a model-agnostic middleware that integrates offline structure-semantic compression with online lightweight evidence purification to yield a compact yet informative database representation. Evaluated on the Spider 2.0-Snow and BIRD benchmarks, the approach reduces input context size by nearly two orders of magnitude while significantly improving both schema linking recall and end-to-end execution accuracy.

Abstract

Recent progress in Text-to-SQL has been driven by stronger language models and prompting strategies, yet performance on real enterprise benchmarks such as Spider 2.0 and BIRD remains far below that on classical academic datasets. We argue that the main bottleneck is no longer reasoning, but database representation. Real databases contain repeated audit columns, large groups of similar tables, opaque identifiers whose meanings are stored only in documentation, and extensive data dictionaries with little query-relevant information. Existing query-aware methods, including schema linking and retrieval-based schema selection, filter this raw context but still operate on redundant and verbose representations. We reformulate the problem as database context compression, a query-agnostic transformation that rewrites schemas, semantic descriptions, and external documentation into a compact representation. We formalize this transformation with the SGCF (Support-Gain Component Factorization) principle, which unifies repeated column extraction, isomorphic table templating, semantic componentization, and evidence purification under a single coverage objective. Based on SGCF, we propose DBCC, a database-side middleware that performs offline structural and semantic compression together with lightweight online evidence purification. DBCC is model-agnostic and can be integrated into existing Text-to-SQL pipelines. On Spider 2.0-Snow and BIRD, DBCC reduces input context by up to two orders of magnitude (from 2.6M to 34.7K tokens on the largest Spider 2.0-Snow subset), improves schema-linking strict recall from 0% to 56.5% under DeepSeek-V3.2 (63.1% under Claude Opus 4.7), and consistently increases end-to-end execution accuracy by 1.8-1.9% over three recent Text-to-SQL systems. Our code is open-sourced at https://github.com/MrBlankness/SchemaCompression.

Problem

• Text-to-SQL
• database context compression
• schema representation
• real-world databases
• redundancy reduction

Innovation

• Database Context Compression
• SGCF
• Text-to-SQL
• Schema Linking
• Model-Agnostic Middleware