Do Agents Need Semantic Metadata? A Comparative Study in Agentic Data Retrieval

2026-05-27

AI Summary

This study investigates whether structured semantic metadata—such as schema.org—remains essential for intelligent agents to achieve reliable and executable data retrieval in the era of large language models (LLMs). By constructing an LLM-as-a-judge evaluation framework, the authors systematically compare semantic-aware agents leveraging such metadata against baseline agents relying solely on open web content, assessing their performance under the FAIR (Findable, Accessible, Interoperable, Reusable) principles. Experimental results demonstrate that semantic agents achieve a 65.7% improvement in overall precision for retrieving FAIR-compliant datasets and a 46.6% gain in identifying results with machine-readable download links. This work provides the first quantitative evidence of the “last-mile utility” of the semantic web ecosystem for executable tasks, underscoring the enduring value of structured metadata even in the age of LLMs.

Abstract

In the era of autonomous agents, machine-actionable data is critical for data-driven workflows. For more than a decade, semantic metadata like schema.org has anchored the FAIR principles (Findable, Accessible, Interoperable, and Reusable) for machine-actionable data and enabled discovery tools like Google Dataset Search. However, the rise of Large Language Models (LLMs) capable of navigating the unstructured web raises a fundamental question: Is semantic metadata still necessary for agentic data discovery, or can agents reliably retrieve actionable data directly from the web? We present a comparative analysis of agentic data retrieval across two distinct environments: a Baseline Agent searching billions of open-web documents, and a Semantic Agent leveraging a corpus of 90 million datasets using schema.org. We deploy an "LLM-as-a-judge" evaluation pipeline, mapped directly to the FAIR principles, to assess the semantic relevance, data accessibility, and computational utility of the retrieved data. Our results reveal a clear divergence. The Semantic Agent excels at retrieving actionable data, achieving a 44.9% higher precision for metadata-rich registries and a 46.6% higher precision for pages with machine-readable downloads among its returned results. Conversely, the Baseline Agent frequently suffers "Last-Mile Utility" failures, retrieving prose-heavy pages (20.1% of results) and portal landing pages (8.5%) rather than actual data pages. While the Baseline Agent achieves higher coverage by answering 40% more questions, the Semantic Agent delivers greater accuracy, achieving 65.7% higher overall precision in retrieving FAIR-compliant datasets. We conclude that while unstructured retrieval supports broad exploratory tasks, structured ecosystems remain the indispensable foundation for reliable, execution-oriented autonomous workflows.

Problem

• semantic metadata
• agentic data retrieval
• FAIR principles
• Large Language Models
• machine-actionable data

Innovation

• semantic metadata
• agentic data retrieval
• FAIR principles
• LLM-as-a-judge
• schema.org