Towards Reliable Generation of Executable Workflows by Foundation Models

2025-09-29

AI Summary

Foundation models (FMs) exhibit low accuracy and high defect rates when generating domain-specific language (DSL) workflows, necessitating extensive manual debugging. Method: We propose a static-analysis–driven closed-loop repair framework. First, we establish a taxonomy of 18 defect classes specific to DSL workflows. Second, we develop Timon—the first static analysis tool tailored for DSL workflows—and integrate it into the FM generation pipeline to automate defect detection, localization, and repair. Contribution/Results: Our analysis reveals that 87.27% of FM-generated workflows contain detectable defects, nine of which are precisely identifiable by Timon. Empirical evaluation demonstrates that our framework significantly improves workflow executability and reliability, advancing end-to-end automation from natural language requirements to executable DSL workflows.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advancements in Foundation Models (FMs) have demonstrated significant progress in comprehending complex natural language to perform intricate tasks. Successfully executing these tasks often requires orchestrating calls to FMs alongside other software components. However, manually decomposing a task into a coherent sequence of smaller, logically aggregated steps, commonly referred to as workflows, demands considerable effort and specialized domain knowledge. While FMs can assist in generating such workflows specified in domain-specific languages (DSLs), achieving accuracy and reliability in this process remains a challenge. This work introduces a framework that leverages static analysis feedback to enable FMs to detect and repair defects in the DSL-based workflows they generate. We begin by presenting the first-ever taxonomy of incidences of defects in FM-generated DSL workflows, categorizing them into 18 distinct types. Furthermore, we observe a high prevalence of defects across FM-generated DSL workflows, with 87.27% of the studied instances containing at least one defect. This, in turn, emphasizes the magnitude of the problem in practice and underscores the necessity for implementing mitigation strategies. Following this, we demonstrate that nine types of these defects can be effectively identified through static analysis of the workflows. For this purpose, we develop Timon, the first-of-its-kind static analyzer specifically designed for FM-generated DSL workflows. Finally, we show that by incorporating feedback from Timon, we can guide Pumbaa, an FM-based tool, to repair the detected defect incidences. By systematically detecting and repairing defects, our work provides a crucial step towards the reliable and automated generation of executable workflows from natural language requirements.

Problem

• Automating workflow generation from natural language using foundation models
• Detecting and repairing defects in domain-specific language workflows
• Improving reliability of executable workflow creation through static analysis

Innovation

• Framework uses static analysis feedback for workflow defect detection
• Introduces Timon static analyzer for FM-generated DSL workflows
• Guides Pumbaa tool to repair detected defects automatically