🤖 AI Summary
This study addresses the “checkpoint storm” problem in shared-storage systems, where concurrent checkpoint writes from multiple training tasks cause severe I/O contention. The authors model training tasks as pulse-coupled oscillators interacting through an anonymous shared resource. Drawing on nonlinear dynamical systems theory, phase map analysis, and extensions of the Kuramoto and Mirollo–Strogatz models—supported by numerical simulations—they demonstrate that under anonymous resource sharing, inter-task phase coupling does not emerge. Consequently, the synchronous state is not an attractor but an unstable fixed point with multiple expanding directions. The analysis proves that anonymous sharing alone cannot induce phase locking or clustering. Although tail-end concurrent write loads remain higher than in fully asynchronous scenarios, genuine coupling arises only when resources are constrained and tasks exhibit heterogeneity.
📝 Abstract
Independent training jobs sharing a storage system write their checkpoints through the same finite bandwidth, and the resulting bursts of correlated I/O are commonly described as a self-reinforcing "checkpoint storm". We formalise the self-reinforcement as phase locking in a population of integrate-and-fire oscillators coupled through a shared resource, and show that within that model it fails. Call a resource anonymous if the rate it delivers to an active user depends on how many users are active and not on which. For identical jobs whose write is shorter than their compute interval, an anonymous resource produces no pairwise coupling at all: the two-job return map of the phase gap is the identity, under storage contention, under a shared power cap and under both, so the two-body interaction on which the Kuramoto and Mirollo-Strogatz frameworks are built is not weak here but absent. Anonymity also freezes the firing order, for any fleet size and any cap, so no trajectory reaches the synchronous state from outside it. What survives is a third-order effect: where all $N$ write windows overlap and the cap does not bind, the map is diagonal in the intervals between consecutive write starts, $a_j \mapsto ((N-j)/j)a_j$, with reciprocal spectrum and unit determinant, making synchrony a fixed point with $\lceil N/2\rceil-1$ expanding directions rather than an attractor. That determinant follows from anonymity and not from fairness: for any anonymous throughput $f$ with $f(n)\le n$ the spectrum becomes $(N-j)f(j)/(j f(N-j))$, whose product is still one. Numerically, a fleet launched at random neither locks nor clusters, and absence of locking is not absence of bursts: the upper tail of the number of concurrent writers stays above its independent-phase value. Heterogeneous jobs behind a binding cap do acquire a genuine pairwise coupling, which is where the statement stops generalising.