FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost

2026-04-27

AI Summary

This work addresses severe computational inefficiencies in large-scale distributed training of sequential recommendation models, where data heterogeneity induces computation bubbles, communication stalls, and GPU resource contention, leading to substantial wasted compute capacity. To mitigate these issues, the authors propose FreeScale, a novel framework that uniquely integrates load-balanced sampling, priority-aware communication scheduling, and an SM-Free communication mechanism. By carefully orchestrating computation and communication, FreeScale eliminates resource conflicts while maximizing their overlap. Evaluated on real-world industrial workloads across 256 H100 GPUs, the approach reduces computation bubbles by up to 90.3% and significantly improves training efficiency with minimal scaling overhead.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems

Abstract

Modern industrial Deep Learning Recommendation Models typically extract user preferences through the analysis of sequential interaction histories, subsequently generating predictions based on these derived interests. The inherent heterogeneity in data characteristics frequently result in substantial under-utilization of computational resources during large-scale training, primarily due to computational bubbles caused by severe stragglers and slow blocking communications. This paper introduces FreeScale, a solution designed to (1) mitigate the straggler problem through meticulously load balanced input samples (2) minimize the blocking communication by overlapping prioritized embedding communications with computations (3) resolve the GPU resource competition during computation and communication overlapping by communicating through SM-Free techniques. Empirical evaluation demonstrates that FreeScale achieves up to 90.3% reduction in computational bubbles when applied to real-world workloads running on 256 H100 GPUs.

Problem

• straggler problem
• computational bubbles
• blocking communication
• resource under-utilization
• sequence recommendation

Innovation

• FreeScale
• straggler mitigation
• communication-computation overlap
• SM-Free communication
• load balancing