Exploring the Semantic Gap in Agentic Data Systems: A Formative Study of Operationalization Failures in Analytical Workflows

2026-07-01

AI Summary

This study addresses the frequent operationalization failures that arise when large language models generate analytical workflows, stemming from a semantic gap between user intent and system-executable actions. Through cross-domain empirical analysis across finance, human resources, and public safety, the authors manually examined 236 analytical intents and their automatically generated workflows, systematically identifying and categorizing five distinct semantic-level failure patterns: comparative anchoring, procedural reasoning, quantitative reasoning, role confusion, and policy anchoring. The findings reveal fundamental limitations in the semantic expressiveness of current data systems and provide both theoretical grounding and practical guidance for improving the reliability of agent-generated analytical workflows.

Abstract

Large language models (LLMs) are increasingly used to generate queries, invoke tools, and construct analytical workflows. Although recent advances have substantially improved workflow generation and execution, the semantic information required to operationalize analytical concepts often lies beyond what is explicitly represented in database schemas and data values. We present a cross-domain formative study of operationalization failures in agent-generated analytical workflows. Across 236 analytical intents spanning finance, human resources, and public safety domains, we identify 153 recurring failures despite successful workflow generation and execution. Our analysis reveals five recurring classes of failures: comparative grounding, process reasoning, quantitative reasoning, role confusion, and policy grounding. These findings suggest a semantic gap between user-level analytical concepts and the information available to workflow-generation systems. More broadly, they raise questions about the admissibility of analytical operations and suggest that future agentic data systems may require richer semantic representations to bridge the gap between analytical intent and executable computation.

Problem

• semantic gap
• operationalization failures
• analytical workflows
• agentic data systems
• large language models

Innovation

• semantic gap
• operationalization failures
• agentic data systems
• analytical workflows
• large language models