SWARM+: Scalable and Resilient Multi-Agent Consensus for Fully-Decentralized Data-Aware Workload Management

2026-03-19

AI Summary

This work proposes a decentralized multi-agent collaborative scheduling framework to address the limitations of centralized schedulers in distributed scientific workflows, including single points of failure, poor scalability, and weak dynamic adaptability. The framework employs a hierarchical consensus architecture supporting up to a thousand agents, integrating a highly fault-tolerant task management mechanism with a data-aware scheduling strategy that jointly considers resource states and data locality. Experimental evaluations on the FABRIC testbed demonstrate that the system scales to 1,000 agents, achieves over 99% task completion rate under single-point failures, incurs no more than a 7.5% performance degradation when 50% of nodes fail, and reduces task scheduling latency by 97–98% compared to baseline approaches.

Technology Category

• Multiagent Systems: Distributed Problem Solving
• Planning, Routing, and Scheduling: Scheduling under Uncertainty
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Distributed scientific workflows increasingly span heterogeneous compute clusters, edge resources, and geo-distributed data repositories. In these environments, a centralized orchestrator is an architectural bottleneck -- introducing a single point of failure, limiting scalability, and constraining adaptability to changing resource availability or failures. Decentralized multi-agent coordination offers a compelling alternative: autonomous agents representing distributed resources collaboratively negotiate workload assignment (e.g., job selection) through peer-to-peer consensus, making decisions based on local compute capacity, data locality, and network conditions. However, scaling such systems for production workloads requires addressing challenges in coordination, resilience, and data-aware optimization. This work presents SWARM+, which builds on our prior work that demonstrated the feasibility of multi-agent decentralized consensus for distributed job selection. SWARM+ addresses three main problems: scalability of consensus for large numbers of agents, resilience of workload management under agent failure, and efficiency of job scheduling for highly distributed resources and data-intensive workloads. For each problem, we propose novel algorithms and evaluate them in the distributed FABRIC testbed. The results show that SWARM+ (a) scales to 1000 distributed agents with nearly equal workload distribution across the hierarchy levels and reduced coordination overhead due to hierarchical consensus, (b) is resilient to agent failures, maintaining >99% job completion rate under single agent failure, and demonstrating graceful system degradation, with at most 7.5% impact under 50% agent failures, and (c) achieves 97-98% improvement over baseline SWARM for both selection time and scheduling latency metrics.

Problem

• decentralized consensus
• multi-agent coordination
• data-aware scheduling
• scalability
• resilience

Innovation

• decentralized multi-agent consensus
• hierarchical consensus
• data-aware scheduling
• resilient workload management
• scalable coordination