Evaluating Retrieval-Augmented Generation Variants for Natural Language-Based SQL and API Call Generation

2026-02-06

AI Summary

This study addresses the challenge of efficiently generating SQL queries and API calls using large language models (LLMs) in enterprise natural language interfaces, particularly in domain-specific settings lacking effective retrieval mechanisms. For the first time, it systematically evaluates standard RAG, Self-RAG, and CoRAG within a real-world enterprise environment—SAP Transactional Banking—across database documentation, API documentation, and hybrid document settings, encompassing 18 experimental configurations. The findings reveal that without retrieval, execution accuracy drops to 0%, while incorporating retrieval boosts accuracy up to 79.30%. Notably, CoRAG achieves a 10.29% exact match rate on hybrid tasks, significantly outperforming standard RAG (7.45%), with pronounced gains in SQL generation. These results validate the superiority of iterative query decomposition strategies in heterogeneous documentation environments.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Enterprise systems increasingly require natural language interfaces that can translate user requests into structured operations such as SQL queries and REST API calls. While large language models (LLMs) show promise for code generation [Chen et al., 2021; Huynh and Lin, 2025], their effectiveness in domain-specific enterprise contexts remains underexplored, particularly when both retrieval and modification tasks must be handled jointly. This paper presents a comprehensive evaluation of three retrieval-augmented generation (RAG) variants [Lewis et al., 2021] -- standard RAG, Self-RAG [Asai et al., 2024], and CoRAG [Wang et al., 2025] -- across SQL query generation, REST API call generation, and a combined task requiring dynamic task classification. Using SAP Transactional Banking as a realistic enterprise use case, we construct a novel test dataset covering both modalities and evaluate 18 experimental configurations under database-only, API-only, and hybrid documentation contexts. Results demonstrate that RAG is essential: Without retrieval, exact match accuracy is 0% across all tasks, whereas retrieval yields substantial gains in execution accuracy (up to 79.30%) and component match accuracy (up to 78.86%). Critically, CoRAG proves most robust in hybrid documentation settings, achieving statistically significant improvements in the combined task (10.29% exact match vs. 7.45% for standard RAG), driven primarily by superior SQL generation performance (15.32% vs. 11.56%). Our findings establish retrieval-policy design as a key determinant of production-grade natural language interfaces, showing that iterative query decomposition outperforms both top-k retrieval and binary relevance filtering under documentation heterogeneity.

Problem

• Retrieval-Augmented Generation
• Natural Language to SQL
• API Call Generation
• Enterprise Systems
• Task Classification

Innovation

• Retrieval-Augmented Generation
• CoRAG
• SQL generation
• API call generation
• enterprise natural language interface