REFORMER: A ChatGPT-Driven Data Synthesis Framework Elevating Text-to-SQL Models

2024-12-18

International Conference on Machine Learning and Applications

AI Summary

To address the weak cross-domain generalization of Text-to-SQL models caused by scarce annotated data in new domains, this paper proposes a fine-tuning-free, ChatGPT-driven data synthesis framework. Methodologically, it integrates zero-shot prompting, SQL semantic modeling, masked generation, and bidirectional paraphrasing. Key contributions include: (1) a retrieval-edit-based question masking and filling mechanism guided by SQL semantic explanations; (2) the first empirical validation and optimization of cycle consistency for generated sample verification; and (3) a novel dual-path ChatGPT rephrasing paradigm that jointly rewrites both natural language questions and their SQL counterparts. Evaluated on multiple cross-domain Text-to-SQL benchmarks, the framework significantly outperforms existing data augmentation methods. Results demonstrate that semantically aligned, bidirectionally rewritten questions substantially enhance model generalization to unseen domains.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

The existing Text-to-SQL models suffer from a shortage of training data, inhibiting their ability to fully facilitate the applications of SQL queries in new domains. To address this challenge, various data synthesis techniques have been employed to generate more diverse and higher quality data. In this paper, we propose REFORMER, a framework that leverages ChatGPT's prowess without the need for additional training, to facilitate the synthesis of (question, SQL query) pairs tailored to new domains. Our data augmentation approach is based on a “retrieve-and-edit” method, where we generate new questions by filling masked question using explanation of SQL queries with the help of ChatGPT. Furthermore, we demonstrate that cycle consistency remains a valuable method of validation when applied appropriately. Our experimental results show that REFORMER consistently outperforms previous data augmentation methods. To further investigate the power of ChatGPT and create a general data augmentation method, we also generate the new data by paraphrasing the question in the dataset and by paraphrasing the description of a new SQL query that is generated by ChatGPT as well. Our results affirm that paraphrasing questions generated by ChatGPT help augment the original data.

Problem

• Addresses shortage of training data for Text-to-SQL models
• Enhances data synthesis for new domains using ChatGPT
• Improves Text-to-SQL performance via retrieval-and-edit augmentation

Innovation

• Leverages ChatGPT for data synthesis
• Uses retrieve-and-edit for question generation
• Applies cycle consistency for validation