PLSQLBench: Benchmarking LLM Systems for Executable Procedural Database Programming

2026-08-16

AI Summary

This study addresses the lack of execution-based evaluation for executable PL/SQL procedural programming in large language models by constructing the first benchmark comprising 2,865 test instances. Integrating multi-turn dialogue with tool-augmented agent techniques, this work fills a critical gap and reveals significant model deficiencies in schema anchoring and flow control. While experiments validate the efficacy of tool augmentation strategies, results indicate that current models still fall short of practical deployment standards. These findings not only highlight existing limitations but also provide clear directions for advancing database programming capabilities in future LLM research.

Abstract

We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based tests. Existing LLM evaluations largely target general-purpose code generation or declarative text-to-SQL, leaving procedural database programming underexplored. PLSQLBench contains 2,865 instances: 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns. The benchmark combines complex schema-grounded tasks over enterprise-style Spider 2 databases, simpler schema-grounded tasks derived from Spider, and MBPP-derived procedural problems, covering varying levels of database grounding and procedural complexity. Experiments with eight LLMs reveal recurring difficulties in schema grounding, PL/SQL dialect fidelity, procedural control flow, exception handling, and cross-turn consistency. Tool-augmented LLM agents improve performance on several schema-grounded evaluations, although substantial gaps remain. These results highlight procedural database programming capabilities not directly assessed by conventional code generation or text-to-SQL benchmarks. Our code is available at https://github.com/oracle-samples/plsqlbench.

Problem

• PL/SQL
• Procedural Database Programming
• LLM Evaluation
• Benchmark
• Executable Code Generation

Innovation

• PLSQLBench
• Procedural Database Programming
• Execution-based Evaluation
• Schema Grounding
• Tool-augmented LLM Agents