Problem
Research questions and friction points this paper is trying to address.
in-context learning
text-to-SQL
accuracy-cost analysis
marginal contribution
pipeline configuration
Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL
🤖 AI Summary
研究通过分析17种配置下五个模块的成本与准确性贡献，解决了现代文本到SQL中成本-准确性量化的问题，揭示了执行反馈精炼的普遍效益及其它模块在特定条件下的作用。
📝 Abstract
Recent advances in in-context learning (ICL) text-to-SQL have substantially improved execution accuracy on public benchmarks by assembling increasingly elaborate pipelines around the base generator, yet existing studies typically report aggregate end-to-end accuracy, without quantifying the marginal accuracy-cost contribution of individual design choices. Consequently, providing a unified, paradigm-level cost-accuracy quantification remains a critical challenge for understanding and configuring modern text-to-SQL. To address this, we instantiate 17 paradigm-level configurations across five recurring modules of the ICL text-to-SQL pipeline under a single controlled implementation, and attribute each paradigm's marginal contribution and incurred cost across all four backbones spanning diverse capability levels and reasoning styles. Our analysis reveals that execution-feedback refinement is the only paradigm whose benefit holds universally at consistently low cost, while most other modules help only under backbone-dependent conditions. Token accounting shows that input demand is more closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior. Cross-module analysis further shows that stacking improves accuracy on most backbones, although how the gains compose varies with backbone capability. We also find that a fixed budget is often better spent engineering a more elaborate pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline. These findings distill into an actionable, cost-aware tiered guideline that transfers to five additional backbones without per-paradigm search.
Innovation
Methods, ideas, or system contributions that make the work stand out.
in-context learning
text-to-SQL
cost-accuracy analysis
execution-feedback refinement
pipeline complexity
🔎 Similar Papers
No similar papers found.
