rush: Scalable Asynchronous Distributed Computing via Shared State in R

2026-06-19

AI Summary

This work addresses the lack of a framework supporting asynchronous, decentralized parallel computation in R, where existing approaches rely on centralized controllers and struggle to meet the demands of efficient shared-state communication required by tasks such as hyperparameter optimization. We propose and implement the *rush* package—the first R-based, Redis-backed framework enabling low-overhead, robust asynchronous shared-state parallelism. *rush* allows worker nodes to asynchronously read and write task data, execute iterations independently, and features automatic failure detection alongside an efficient caching mechanism. It provides high-level APIs for managing task lifecycles and integrates seamlessly with the *mlr3*, *bbotk*, and *mlr3tuning* ecosystems. Empirical evaluation demonstrates that its Asynchronous Decentralized Bayesian Optimization (ADBO) efficiently conducts LightGBM hyperparameter searches across four datasets using 448 workers, achieving per-task communication overhead below one millisecond.

Abstract

Many algorithms in statistics and machine learning can be parallelized in an asynchronous manner where workers need to communicate through shared state rather than execute independent tasks dispatched by a central controller. Especially in modern hyperparameter optimization and parallel black-box optimization with expensive objectives, this decentralized approach has become widespread, and several Python frameworks adopt it (e.g., Optuna, DeepHyper, and Hyperopt). However, all popular R packages for parallel computing follow a centralized controller-worker architecture that does not support this pattern. We present rush, an R package that provides a shared-state coordination layer for asynchronously parallelized iterative algorithms. rush uses a Redis database as a shared key-value store: workers read and write task data through the database and independently execute their own loops. The package provides a high-level API for managing tasks and their lifecycle, featuring sub-millisecond per-task overhead, robust error handling with automatic detection of lost workers, and efficient caching. rush optionally integrates with the mlr3 ecosystem, powering asynchronous optimization in the bbotk and mlr3tuning packages. We demonstrate the practical utility of rush by implementing asynchronous decentralized Bayesian optimization (ADBO) and benchmarking it on hyperparameter optimization of LightGBM across four datasets using 448 workers.

Problem

• asynchronous distributed computing
• shared state
• R programming
• hyperparameter optimization
• black-box optimization

Innovation

• asynchronous distributed computing
• shared state
• R package
• Redis
• decentralized optimization