tensor parallelism

To address high per-node memory pressure and substantial intra-node communication overhead in distributed pre-training of large-scale models, this paper proposes Subnet Data Parallel (SDP): each worker node independently trains a structured, compact subnetwork, eliminating activation transmission across pipeline stages. SDP integrates stochastic block dropping with width-wise subnetwork construction to ensure uniform parameter coverage and gradient alignment across distributed workers. Its communication bandwidth requirement is comparable to or lower than that of standard all-reduce operations. Experiments demonstrate that SDP reduces GPU memory consumption by 20–40% without compromising model accuracy, while preserving convergence properties and significantly improving training efficiency for large models.

To address prominent computation and communication bottlenecks in distributed training of large language model (LLM)-based recommender systems, this paper proposes a hybrid parallel architecture integrating tensor parallelism, pipeline parallelism, and asynchronous data parallelism. It innovatively introduces an adaptive load-balancing mechanism and an efficient sparse gradient aggregation communication framework—supporting both gradient compression and sparsification. The design ensures scalability and robustness under online deployment. Experiments on real-world recommendation datasets demonstrate that the proposed approach achieves over 30% improvement in training throughput and approximately 20% higher GPU resource utilization compared to baseline methods. These results validate its strong scalability and training stability, establishing significant advances in efficient large-scale LLM recommender training.

This study addresses the heterogeneous inter-GPU communication patterns induced by tensor parallelism, pipeline parallelism, and their hybrid variants in distributed large language model (LLM) inference—patterns that critically impact end-to-end latency, network overhead, and service-level objective (SLO) compliance. Method: We propose an integrated “measurement-driven analysis–analytical modeling–experimental validation” methodology, establishing a fine-grained communication behavior characterization framework to quantify communication latency and bandwidth consumption across varying sequence lengths, model scales, and hardware topologies. Contribution/Results: We identify three key trade-offs: (i) tensor parallelism enables low-latency short-sequence responses but causes high network saturation; (ii) pipeline parallelism reduces per-transfer volume yet introduces substantial bubble latency; and (iii) hybrid parallelism requires dynamic, load-aware tuning of communication-computation overlap. Our findings provide interpretable, reusable theoretical foundations and practical guidelines for parallelism selection, communication optimization, and SLO-aware deployment in production-grade LLM inference systems.

Modern large-scale distributed training faces sharply diminishing returns in hardware scaling: as GPU counts reach thousands, communication overhead dominates performance bottlenecks, rendering conventional parallelism strategies—data, tensor, and pipeline parallelism—suboptimal. Method: Leveraging real-world LLM training workloads, this project establishes an empirical analytical framework spanning diverse model scales, hardware configurations, and parallelization strategies. It quantifies the nonlinear relationship between accelerator count and performance gain, precisely identifying critical inflection points across model, data, and compute scaling dimensions. Contribution/Results: We discover that low-communication “suboptimal” strategies become optimal at extreme scale; we empirically determine hardware selection criteria, cluster topology requirements, and optimal parallelism combinations for training billion-parameter models. Our findings provide actionable, deployment-ready optimization guidelines for trillion-parameter LLM training infrastructures.

Static parallelization strategies in large-scale distributed neural network training suffer from poor resource adaptability, leading to efficiency bottlenecks. To address this, we systematically evaluate the performance boundaries of data parallelism, model parallelism, and hybrid parallelism, and propose a dynamic, topology- and resource-aware scheduling algorithm. This algorithm enables online switching of parallelization strategies within a hybrid parallel framework during training, jointly optimizing communication overhead, computational load balance, and memory constraints. On the CIFAR-100 image classification benchmark, hybrid parallelism achieves a 3.2× speedup over single-GPU training with no accuracy degradation; integrating our adaptive scheduler further improves end-to-end training efficiency by 18%. To the best of our knowledge, this work is the first to incorporate dynamic strategy switching into the hybrid parallel training pipeline, establishing a scalable new paradigm for efficient large-model training in heterogeneous resource environments.

This work addresses the limited scalability of tensor parallelism in large model online inference, where non-scalable overheads hinder near-linear cluster performance scaling. The authors propose Albireo, a system that eliminates such bottlenecks without modifying model architecture by overlapping scheduling with computation, employing sequence-parallel sampling, and optimizing KV cache management. Albireo further introduces the concept of “empirically optimal tensor parallelism degree” to guide parallelism strategy selection. Experimental results demonstrate that, compared to vLLM, Albireo achieves up to 1.9× higher throughput, 48% lower latency, 28% improved GPU utilization, and 54% reduced energy consumption, with a twofold throughput gain observed in production environments.

This work addresses the GPU memory bottleneck in Transformer models caused by high parameter and activation memory demands during training and inference. The authors propose a novel parallelism strategy that integrates tensor parallelism (TP) and sequence parallelism (SP) along the same device axis, enabling each device to simultaneously shard both model weights and input sequences. By leveraging broadcast-based weight sharding with key-value exchange in attention layers and ring-based weight passing with local accumulation in gated MLPs, the method achieves dual compression of both parameter and activation memory. This approach significantly reduces per-device memory consumption, outperforming conventional TP, SP, and their hybrid variants. It demonstrates superior hardware adaptability and scaling efficiency under long-context and memory-constrained settings, while seamlessly integrating with pipeline and expert parallelism.

This work addresses the limited flexibility in distributed programming for large language model scaling and the inefficiency of existing tensor compilers in handling the complex memory hierarchies of heterogeneous clusters. To overcome these challenges, the authors propose a scalable block-level compiler featuring a novel three-tier hierarchical abstraction—Core, Device, and Task—that uniformly supports diverse parallelization strategies, automatically optimizes intra- and inter-node communication, and enables efficient code generation across both NVIDIA and AMD platforms. When integrated into vLLM, the compiler achieves 5%–30% end-to-end inference speedup and over 10% improvement in training model FLOPs utilization (MFU), translating to approximately 500,000 GPU hours saved per month. The system has been deployed in enterprise settings, delivering over 20% inference performance gains.

Sparse tensor algebra poses significant challenges for efficient load balancing on parallel systems due to its irregular structure, strong data dependencies, and skewed distributions. This work proposes the first provably load-balanced partitioning algorithm capable of handling arbitrary numbers of operands and multi-dimensional hierarchical sparsity patterns. The algorithm is integrated into a sparse tensor algebra compilation framework that automatically generates high-performance parallel kernels for both multicore CPUs and GPUs. By overcoming the limitations of conventional parallel merge strategies, the approach produces code that matches or exceeds the performance of hand-optimized libraries such as Intel MKL and cuSPARSE, achieving geometric mean speedups of 0.73–3.4× across diverse hardware platforms. Moreover, it substantially outperforms existing general-purpose parallelization strategies, with geometric mean speedups ranging from 2.0× to 6.4×.