apache airflow

To address data scarcity, weak infrastructure, and challenges in processing heterogeneous multi-source data for air quality monitoring in resource-constrained regions, this paper designs and implements AirQo—a cloud-native data pipeline. We propose a modular ETL architecture featuring a decoupled ingestion layer, AI-driven automatic sensor calibration using lightweight ML models, and fault-tolerant mechanisms for network/power outages, all integrated within a fully observable end-to-end framework. Real-time streaming is decoupled via Apache Kafka, batch and stream workflows are orchestrated with Apache Airflow, and analytical workloads are supported by BigQuery. Deployed across 400+ low-cost sensors in Africa, AirQo processes over ten million records monthly with demonstrated stability. Evaluation shows a 32% reduction in calibration error and a 45% decrease in computational resource overhead, validating its scalability, robustness, and cross-regional reusability.

In dynamic scientific workflows, the decoupling of task scheduling from data movement often assigns tasks to nodes lacking local input data, causing network congestion and execution delays. To address this, we propose the first workflow-aware joint scheduling framework that enables “data-readiness-driven” task placement via predictive pre-staging of intermediate data replicas, supporting dynamic execution plans. Our approach integrates speculative data pre-staging, dependency-aware scheduling, and lightweight storage management, implemented on a Nextflow+Kubernetes prototype. Experiments across 16 synthetic and real-world workflows demonstrate significant reductions in total completion time—up to 94.5% (synthetic) and 53.2% (real), with only bounded, transient storage overhead. The core contribution is the first holistic co-optimization of scheduling and data movement for dynamic workflows, breaking the traditional separation between these concerns.

This work addresses the limitations of traditional workflow platforms, which rely on static, pre-defined processes and struggle to accommodate the dynamic data integration demands of distributed systems. To overcome this, the authors propose a configuration-driven runtime orchestration framework that dynamically constructs execution graphs at request time through dependency-aware scheduling and parallel task execution, thereby circumventing the constraints of fixed workflows. This approach enables rapid adaptation to evolving integration scenarios without requiring system redeployment, significantly reducing latency. Empirical evaluation in a real-world Customer 360 enterprise use case demonstrates that the framework offers substantial advantages in flexibility, scalability, and efficient data aggregation compared to conventional solutions.

Existing agent-based AI workflow frameworks struggle to balance scalability and reproducibility due to fragmented data orchestration, high serialization overhead, and non-deterministic execution. This work proposes modeling agent workflows as operator abstractions and introduces a unified distributed runtime that enables efficient interoperability among preprocessing, embedding, and vector retrieval through a zero-copy data plane built on Apache Arrow and Cylon. By incorporating resource-deterministic scheduling and asynchronous batching, the system achieves, for the first time, deterministic and scalable execution of agent workflows under high-performance computing paradigms. Experimental results demonstrate that, while maintaining comparable large language model generation throughput, the system attains up to a 4.64× pipeline speedup and a 2.8× improvement in embedding write performance.

This work addresses the limitation of existing distributed data pipeline systems, which require users to explicitly define complete workflow graphs, by proposing a unified planning and scheduling framework that automatically constructs end-to-end persistent pipelines from implicit goal declarations alone. The approach introduces, for the first time, a numeric-domain-independent planner into the context of persistent scheduling, integrating workflow and resource graph modeling, numeric planning, and network interface scheduling to achieve full automation. Experimental results demonstrate the feasibility and scalability of the method: under a single-machine constraint of one hour of CPU time and 30 GB of memory, the system successfully scheduled a linear pipeline spanning eight sites and comprising fourteen components.

Existing visual analytics workflows are predominantly described in unstructured textual form, hindering systematic comparison, reuse, and practical guidance. This work proposes ATWL, a formal, declarative language for modeling visual analytics workflows through a modular ontology grounded in eight artifact types and standardized intents. For the first time, this approach enables structured, machine-interpretable representations of such workflows. Leveraging large language models, the authors automatically extract workflows from academic papers to construct a reusable repository comprising 17 annotated instances. Empirical evaluation demonstrates that ATWL effectively uncovers cross-workflow structural patterns and yields more compact, structured, and extensible analytical recommendations than original narrative descriptions, thereby facilitating efficient in-context reuse and adaptation.

Automated construction of executable ELT pipelines faces significant challenges, including ambiguous user intent, unreliable tool generation, and a lack of execution guarantees in outputs. This work proposes kRAIG, a natural language–driven AI agent that leverages a novel ReQuesAct interaction framework to explicitly clarify user intent. By integrating retrieval-augmented component composition with a multi-stage LLM-based verification mechanism, kRAIG automatically translates high-level instructions into production-grade Kubeflow pipelines. Experimental results demonstrate that kRAIG achieves a threefold improvement in data extraction and loading success rates and a 25% increase in transformation accuracy compared to existing approaches, substantially enhancing the reliability, executability, and practicality of automated data engineering pipelines.

Current large language model (LLM) agent workflow systems rely on predefined templates and shallow matching, limiting their ability to capture deep semantic relationships and generalize effectively. This work proposes GraphFlow, a novel framework that introduces a unified graph structure—wGraph—as a shared workflow substrate, enabling adaptive, semantics- and constraint-aware dynamic workflow generation. To enhance inference efficiency, GraphFlow incorporates a structure-aware key-value (KV) cache management mechanism. Through the synergistic optimization of graph representation learning, dynamic workflow instantiation, and structured caching, GraphFlow achieves an average performance gain of 4.95 percentage points across five benchmark datasets while reducing memory consumption by approximately fourfold.

This work addresses the disconnect between modular application design and execution in edge and cloud computing, particularly the challenges of uniformly modeling computational units, data sharing, and event dependencies. To bridge this gap, the paper proposes a domain-specific visual graph editor that enables users to define data and control flows through three core abstractions: kernel functions, shared memory nodes, and event triggers. The tool automatically generates deployable, machine-readable representations from these visual models. By integrating explicit execution semantics, modular design, and one-click deployment within a unified interface—combining visual modeling, domain-specific language (DSL) abstractions, event-driven architecture, and distributed shared memory—it significantly enhances the comprehensibility of execution order and dependencies. Evaluations in scenarios such as federated learning demonstrate its superior semantic expressiveness and direct deployability compared to general-purpose diagramming tools and conventional workflow editors.