JANUS: Resilient and Adaptive Data Transmission for Enabling Timely and Efficient Cross-Facility Scientific Workflows

2025-06-20

AI Summary

To address bottlenecks in wide-area, large-scale data transfer for cross-institutional scientific workflows—including excessive bandwidth consumption, inefficient TCP packet-loss recovery, and high overhead of conventional fault-tolerance mechanisms—this paper proposes a UDP-based elastic and adaptive transfer framework. The framework innovatively integrates error-bounded lossy compression (e.g., SZ), dynamically tunable fountain-code erasure coding, and a real-time network-aware integer programming optimization model. This enables on-demand trade-offs between timeliness and fidelity, as well as online, self-adaptive transmission policy adjustment. Experimental evaluation demonstrates that, compared to Globus and standard TCP-based approaches, the framework achieves up to a 3.2× improvement in transfer throughput, accelerates packet-loss recovery by up to 5.8× under bandwidth fluctuations, and ensures controllable, mathematically verifiable data fidelity.

Technology Category

• Data Mining & Knowledge Management: Data Compression
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Distributed Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Data transparency and provenance

Abstract

In modern science, the growing complexity of large-scale projects has increased reliance on cross-facility workflows, where institutions share resources and expertise to accelerate discovery. These workflows often involve transferring massive data over wide-area networks. While high-speed networks like ESnet and data transfer services like Globus have improved data mobility, challenges remain. Large data volumes can strain bandwidth, TCP suffers from retransmissions due to packet loss, and traditional fault-tolerance methods like erasure coding introduce significant overhead. This paper presents JANUS, a resilient and adaptive data transmission approach for cross-facility scientific workflows. JANUS uses UDP, integrates erasure coding for fault tolerance, and applies error-bounded lossy compression to reduce overhead. This design enables users to balance transmission time and accuracy based on specific needs. JANUS also adapts coding parameters to real-time network conditions and uses optimization models to determine ideal configurations. Experiments show that JANUS significantly improves data transfer efficiency while preserving fidelity.

Problem

• Improving data transfer efficiency in cross-facility workflows
• Reducing overhead from fault-tolerance methods like erasure coding
• Adapting to real-time network conditions for optimal performance

Innovation

• UDP-based resilient data transmission
• Error-bounded lossy compression technique
• Real-time adaptive coding parameters