Strengthening LargeRDFBench for Interoperable Federated SPARQL Evaluation

2026-08-31

AI Summary

"This study addresses data quality issues and non-reproducible expected results in some datasets of the LargeRDFBench benchmark, which impact the accuracy of federated SPARQL query engine evaluations. The project introduces a reproducible data cleansing process and re-encodes expected results into W3C standard formats, ensuring compatibility with RDF-compliant engines. By aligning the benchmark with RDF standards, the scope of fairly and reproducibly comparable engines is expanded. Notably, the work explores for the first time ASK and COUNT-based selection strategies within the FedX algorithm. Utilizing rigorous RDF parser validation and W3C SPARQL 1.1 Query Results JSON Format, this study ensures that each affected dataset can now be correctly parsed under strict conditions, enhancing the verifiability and reliability of benchmarking outcomes."

Abstract

LargeRDFBench is one of the most comprehensive benchmarks for evaluating federated SPARQL query engines, combining real, interlinked datasets with a rich query suite that has made it a reference point for the community. Evaluations of federated engines are published by comparing engine results against the benchmark's expected results, so those expected results must themselves be reproducible. Moreover, several of its data dumps violate the RDF specifications, so only engines that parse RDF leniently can host them, and its expected results are distributed in an ad hoc format. We identify, categorize and repair these data-quality issues with a reproducible cleaning pipeline, producing standards-conformant serializations of every affected dataset. Furthermore, we re-encode the benchmark's expected results in the W3C SPARQL 1.1 Query Results JSON Format and correct their discrepancies. Every dataset now parses under strict RDF parsers, and the expected results are machine-verifiable through a standard format, extending the benchmark's reach to the full range of conformant engines while staying faithful to the original data. Reproducing the expected results end-to-end with an independent implementation uncovers corruption in the published reference, and discrepancies between our results and the original ones, some not trivial to resolve, others open questions. We further perform a preliminary comparison, not previously explored, of ASK- and COUNT-based source selection in the FedX algorithm. This work strengthens an already valuable community resource by aligning its artifacts with the RDF standards, broadening the set of engines that can be fairly and reproducibly compared. We also raise the question of how the results of federated queries under automatic source selection can be made reproducible.

Problem

• federated SPARQL
• benchmark
• data quality
• reproducibility
• RDF standards

Innovation

• RDF Standards Compliance
• Federated SPARQL Query Engines
• Reproducible Cleaning Pipeline
• Standardized Expected Results
• Source Selection Strategies