Problem
Research questions and friction points this paper is trying to address.
NL2SHACL
Benchmark
SHACL
Natural Language Requirements
Semantic Equivalence
NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation
🤖 AI Summary
为解决非专业人员难以编写SHACL的问题，提出NL2SHACL-Bench基准套件，通过自然语言到SHACL的转换方法，并评估了四种先进大语言模型的表现。
📝 Abstract
SHACL is a core technology for validating the conformance of RDF knowledge graphs (KGs). Yet, authoring SHACL shapes requires technical expertise that most domain experts lack. Translating natural language requirements into SHACL (NL2SHACL) would lower this barrier. However, there is no dedicated benchmark for NL2SHACL, and evaluating generated shapes requires methods beyond string comparison, as semantically equivalent shapes can differ in serialisation and structure. To tackle these challenges, we present NL2SHACL-Bench, a benchmark suite for natural language to SHACL translation. Using NL2SHACL-Bench, we evaluate four state-of-the-art large language models (LLMs) for this task. Our results show that current LLMs are highly capable of generating syntactically valid SHACL, but still struggle to produce semantically equivalent constraints for complex logical and structural patterns. This indicates that NL2SHACL-Bench provides a meaningful basis for measuring advances in the NL2SHACL state of the art.
Innovation
Methods, ideas, or system contributions that make the work stand out.
NL2SHACL
Benchmark
Semantic Equivalence
Large Language Models
SHACL
🔎 Similar Papers
No similar papers found.
