Faster Distributed Inference-Only Recommender Systems via Bounded Lag Synchronous Collectives

2025-12-22

AI Summary

In DLRM distributed inference, sparse feature embedding lookups trigger an all-to-allv (a2av) communication pattern, whose severe synchronization blocking—caused by inter-process latency variance—becomes a critical performance bottleneck. To address this, we propose Bounded-Lag Synchronization (BLS), the first accuracy-preserving asynchronous collective communication paradigm for inference: it permits straggling processes to lag by entire iterations without stalling faster ones. Built upon a custom PyTorch Distributed backend and a specialized BLS-enabled a2av operator, our lightweight DLRM inference framework eliminates unnecessary synchronization overhead. Experiments under heterogeneous deployments demonstrate substantial improvements: throughput increases significantly, end-to-end latency decreases markedly, and in optimal cases, communication wait time is fully overlapped—achieving an end-to-end speedup of over 1.8×.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recommender systems are enablers of personalized content delivery, and therefore revenue, for many large companies. In the last decade, deep learning recommender models (DLRMs) are the de-facto standard in this field. The main bottleneck in DLRM inference is the lookup of sparse features across huge embedding tables, which are usually partitioned across the aggregate RAM of many nodes. In state-of-the-art recommender systems, the distributed lookup is implemented via irregular all-to-all (alltoallv) communication, and often presents the main bottleneck. Today, most related work sees this operation as a given; in addition, every collective is synchronous in nature. In this work, we propose a novel bounded lag synchronous (BLS) version of the alltoallv operation. The bound can be a parameter allowing slower processes to lag behind entire iterations before the fastest processes block. In special applications such as inference-only DLRM, the accuracy of the application is fully preserved. We implement BLS alltoallv in a new PyTorch Distributed backend and evaluate it with a BLS version of the reference DLRM code. We show that for well balanced, homogeneous-access DLRM runs our BLS technique does not offer notable advantages. But for unbalanced runs, e.g. runs with strongly irregular embedding table accesses or with delays across different processes, our BLS technique improves both the latency and throughput of inference-only DLRM. In the best-case scenario, the proposed reduced synchronisation can mask the delays across processes altogether.

Problem

• Reduces communication bottleneck in distributed recommender systems
• Enables asynchronous embedding lookups with bounded lag
• Improves latency and throughput for unbalanced inference workloads

Innovation

• Bounded lag synchronous alltoallv operation
• Parameter allowing slower processes to lag
• Preserves accuracy in inference-only DLRM