orchestration

This work addresses the limitations of traditional workflow platforms, which rely on static, pre-defined processes and struggle to accommodate the dynamic data integration demands of distributed systems. To overcome this, the authors propose a configuration-driven runtime orchestration framework that dynamically constructs execution graphs at request time through dependency-aware scheduling and parallel task execution, thereby circumventing the constraints of fixed workflows. This approach enables rapid adaptation to evolving integration scenarios without requiring system redeployment, significantly reducing latency. Empirical evaluation in a real-world Customer 360 enterprise use case demonstrates that the framework offers substantial advantages in flexibility, scalability, and efficient data aggregation compared to conventional solutions.

This study addresses the challenge of automating workflows in complex industries—such as logistics, healthcare, and construction—where processes are fragmented across heterogeneous tools and involve multi-party collaboration. The work proposes orchestration as a core abstraction to enable effective automation by dynamically coordinating multi-step tasks, enforcing domain-specific constraints, managing human approvals, and integrating legacy systems. It introduces the novel concept of “orchestration bottlenecks” and develops a theoretical framework that unifies multi-agent systems, workflow modeling, constraint reasoning, and human–AI collaboration, while exposing critical gaps in current multi-agent approaches at the orchestration level. Based on distinct sources of operational friction across domains, the paper advocates for targeted architectural safeguards—such as constraint enforcement or explainability—and phased implementation strategies to provide actionable pathways for automation in complex operational environments.

To address the challenge of efficiently orchestrating and intelligently managing complex, dynamic workflows in large-scale distributed scientific computing, this paper proposes an integrated intelligent workflow system that unifies task scheduling, data movement, and adaptive decision-making. The system supports data-aware execution, conditional logic, and programmable directed acyclic graphs (DAGs), operating in both template-driven and “function-as-a-task” modes. It adopts a modular, message-driven architecture and deeply integrates mainstream middleware—including PanDA and Rucio—while incorporating distributed hyperparameter optimization and AI-assisted modeling. Its cross-experiment, cross-platform design significantly enhances scalability and reproducibility. Deployed in major scientific projects—including ATLAS, the Rubin Observatory, and the Electron-Ion Collider—the system enables high-throughput execution of heterogeneous tasks and reduces operational overhead by over 30%.

This work addresses the unreliability of developer productivity dashboards, which often stems from ad hoc scripts that introduce undetected silent data gaps, eroding organizational trust. To resolve this, we propose a robust ELT pipeline grounded in DAG-based orchestration and the Medallion architecture, decoupling data extraction from transformation to preserve the immutability of raw data. Our approach introduces a state-driven dependency scheduling mechanism and, for the first time, treats metric pipelines as production-grade distributed systems. We emphasize the critical role of immutable raw history in enabling reliable metric redefinition. This methodology significantly enhances data reliability and freshness while effectively eliminating silent failures, thereby restoring organizational confidence in DevOps metrics.

The workflow management system (WMS) domain has long suffered from a lack of standardized terminology, impeding objective system selection, reproducible evaluation, and interoperability. To address this, we propose the first “Five-Axis Standardized Terminology Framework,” systematically characterizing WMS capabilities across five dimensions: workflow characteristics, orchestration, composition, data management, and metadata capture. Leveraging conceptual modeling, cross-system feature analysis, expert consensus workshops, and empirical classification, we establish the first community-endorsed terminology taxonomy. We apply this framework to perform structured capability mapping and classification of 23 widely adopted WMSs. Our framework shifts WMS evaluation from anecdotal, experience-driven practices toward comparable, reproducible, and evidence-based decision-making. It enhances transparency in system selection and lays a foundational semantic basis for interoperability across heterogeneous scientific workflows.

This work addresses the limitations of existing agent orchestration frameworks, which rely on external schedulers and incur substantial context overhead, require state-of-the-art large language models, and risk exposing proprietary workflows. To overcome these issues, the authors propose compiling multi-node agent workflows—comprising up to 55 nodes—directly into the weights of a small fine-tuned language model, thereby creating what they term “underground agents.” This approach provides the first systematic demonstration that complex workflows can be effectively internalized within model parameters. By integrating structured workflow representations, task-specific knowledge injection, and decision-hub modeling, the method achieves performance comparable to leading models on tasks such as travel booking, Zoom customer support, and insurance claims processing, while reducing inference costs by two orders of magnitude and substantially diminishing reliance on conventional orchestration frameworks.

This work proposes a novel approach to agent orchestration by embedding complete task logic directly into the system prompt, enabling large language models (LLMs) to self-orchestrate complex, multi-step workflows without external controllers. Evaluated across three real-world domains—travel booking, Zoom technical support, and insurance claims processing—with task graphs comprising 14 to 55 nodes, the method demonstrates significant advantages over prevailing external frameworks such as LangGraph. Using LLM-as-judge automated evaluation, in-context prompting achieves task scores of 4.53–5.00 on a 5-point scale and reduces failure rates by 12.5%, 8.5%, and 12% respectively. This study provides the first systematic evidence that state-of-the-art LLMs can autonomously and reliably execute structured, programmatic tasks at scale.

Distributed scientific workflows often exhibit highly unpredictable behavior, making it challenging to satisfy Quality-of-Service (QoS) constraints such as execution time or resource limits. To address this issue, this work proposes QoSFlow, an interpretable modeling approach based on statistical sensitivity analysis that partitions the configuration space into regions of similar behavioral characteristics, enabling efficient and accurate QoS-aware scheduling. By integrating performance modeling with analytical reasoning, QoSFlow avoids the need for exhaustive empirical testing. Experimental evaluation on three representative workflows demonstrates that configurations recommended by QoSFlow outperform those from the best heuristic methods by an average of 27.38%, with consistently stable real-world performance. This study presents the first interpretable, high-precision optimization framework for QoS in distributed scientific workflows.

This study addresses the challenges of high latency, unstable concurrency, and security risks faced by large language model (LLM) agents in automating asset lifecycle management within Industry 4.0. The authors propose a Plan-then-Execute architecture that generates verifiable workflow graphs and integrates a topology-aware parallel scheduling mechanism to enable controlled inference overlap while ensuring functional correctness and security. Key technical contributions include topological-sort-based multi-agent scheduling, structured context pruning, dependency-aware concurrency control, and graceful degradation under fault injection. Evaluated on the AssetOpsBench benchmark, the system reduces median end-to-end latency by 1.6× (up to 1.8× for highly parallel tasks) and cuts inference overhead by approximately 30% through context pruning, all while maintaining stable task completion rates and output quality.

This work addresses the growing challenge in modern scientific research where complex infrastructure management, authentication, and deployment processes divert focus from core scientific discovery. To this end, the authors propose Sci-Orchestra, a cloud-native, Kubernetes-based hierarchical orchestration framework that automates experimental workflows through API-driven mechanisms, enabling secure authentication, resource scheduling, and scalable deployment across heterogeneous high-performance computing environments. Sci-Orchestra introduces an autonomous service marketplace to facilitate cross-institutional collaboration and adopts a “black-box” interoperability model that promotes integration of academic, industrial, and research tools while safeguarding intellectual property. By significantly lowering technical barriers, the framework accelerates the transition of research prototypes into production-grade applications, thereby advancing the emerging paradigm of Science as a Service (SciaaS).