BLAST: Benchmarking LLMs with ASP-based Structured Testing

2026-04-24

AI Summary

This work addresses the absence of dedicated evaluation benchmarks for assessing the accuracy of Answer Set Programming (ASP) code generated by large language models (LLMs). To bridge this gap, the authors propose BLAST—the first structured evaluation framework specifically designed for ASP code generation by LLMs—encompassing ten categories of graph-theoretic problems and eight prominent LLMs. BLAST introduces a semantics-aware test suite grounded in ASP formalism and two novel semantic-level evaluation metrics, thereby filling a critical void in evaluating LLM capabilities within declarative programming paradigms. Systematic experiments not only characterize the current performance of state-of-the-art LLMs on ASP generation tasks but also uncover their fundamental limitations in handling the structural and logical nuances inherent to ASP.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Knowledge Representation and Reasoning: Logic Programming
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across a broad spectrum of tasks, including natural language understanding, dialogue systems, and code generation. Despite evident progress, less attention has been paid to their effectiveness in handling declarative paradigms such as Answer Set Programming (ASP), to date. In this paper we introduce BLAST: The first dedicated benchmarking methodology and associated dataset for evaluating the accuracy of LLMs in generating ASP code. BLAST provides a structured evaluation framework featuring two novel semantic metrics tailored to ASP code generation. The paper presents the results of an empirical evaluation involving ten well-established graph-related problems from the ASP literature and a diverse set of eight state-of-the-art LLMs.

Problem

• Large Language Models
• Answer Set Programming
• Code Generation
• Benchmarking
• Declarative Paradigms

Innovation

• Answer Set Programming
• Large Language Models
• structured testing
• semantic metrics
• benchmarking