ETM: Modern Insights into Perspective on Text-to-SQL Evaluation in the Age of Large Language Models

2024-07-10

AI Summary

Existing Text-to-SQL evaluation metrics—Execution Accuracy (EXE) and Exact Set Match (ESM)—suffer from fundamental limitations: EXE overlooks syntactic errors, leading to inflated correctness estimates, while ESM’s rigid lexical matching underestimates semantically equivalent queries. Both yield severe misjudgments when evaluating diverse SQL outputs generated by large language models (LLMs). To address this, we propose Enhanced Tree Matching (ETM), the first metric integrating abstract syntax tree (AST) structural alignment with subtree-level semantic equivalence assessment. ETM incorporates schema-aware normalization and configurable similarity thresholds, thereby transcending traditional binary matching paradigms. Experiments across nine state-of-the-art LLMs demonstrate that ETM reduces false positive rate from 23.0% to 0.3% and false negative rate from 28.9% to 2.7%, significantly enhancing evaluation robustness and granularity. The implementation is publicly available.

Technology Category

• Machine Learning: Evaluation and Analysis
• Search and Optimization: Evaluation and Analysis
• Natural Language Processing: Information Extraction

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The task of Text-to-SQL enables anyone to retrieve information from SQL databases using natural language. While this task has made substantial progress, the two primary evaluation metrics -- Execution Accuracy (EXE) and Exact Set Matching Accuracy (ESM) -- suffer from inherent limitations that can misrepresent performance. Specifically, ESM's rigid matching overlooks semantically correct but stylistically different queries, whereas EXE can overestimate correctness by ignoring structural errors that yield correct outputs. These shortcomings become especially problematic when assessing outputs from large language model (LLM)-based approaches without fine-tuning, which vary more in style and structure compared to their fine-tuned counterparts. Thus, we introduce a new metric, Enhanced Tree Matching (ETM), which mitigates these issues by comparing queries using both syntactic and semantic elements. Through evaluating nine LLM-based models, we show that EXE and ESM can produce false positive and negative rates as high as 23.0% and 28.9%, while ETM reduces these rates to 0.3% and 2.7%, respectively. We release our ETM script as open source, offering the community a more robust and reliable approach to evaluating Text-to-SQL.

Problem

• Limitations of current Text-to-SQL evaluation metrics
• Inaccurate performance assessment by EXE and ESM
• Proposal of Enhanced Tree Matching (ETM) metric

Innovation

• Enhanced Tree Matching metric
• Combines syntactic semantic elements
• Reduces false positive negative rates