Compression-Induced Communication-Efficient Large Model Training and Inferencing

2025-08-01

AI Summary

To address the high energy consumption and communication overhead of tensor parallelism in large-model distributed training, this paper proposes *Phantom Parallelism*, a technique specifically targeting feed-forward network (FFN) architectures. It reconstructs forward and backward operators to enable efficient training of compact “phantom models” on fewer GPUs, achieving performance comparable to large-scale tensor parallelism. The method employs custom autograd operations to build an end-to-end training pipeline and integrates compression-induced communication optimizations. Experiments on 256 GPUs demonstrate that Phantom Parallelism reduces FFN training energy consumption by approximately 50% compared to conventional tensor parallelism; under equivalent model loss, it significantly decreases required GPU count and substantially improves energy efficiency. Its core innovation lies in decoupling model scale from hardware parallelism—achieving, for the first time, systematic energy-cost reduction for distributed FFN training without compromising accuracy.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Energy efficiency of training and inferencing with large neural network models is a critical challenge facing the future of sustainable large-scale machine learning workloads. This paper introduces an alternative strategy, called phantom parallelism, to minimize the net energy consumption of traditional tensor (model) parallelism, the most energy-inefficient component of large neural network training. The approach is presented in the context of feed-forward network architectures as a preliminary, but comprehensive, proof-of-principle study of the proposed methodology. We derive new forward and backward propagation operators for phantom parallelism, implement them as custom autograd operations within an end-to-end phantom parallel training pipeline and compare its parallel performance and energy-efficiency against those of conventional tensor parallel training pipelines. Formal analyses that predict lower bandwidth and FLOP counts are presented with supporting empirical results on up to 256 GPUs that corroborate these gains. Experiments are shown to deliver ~50% reduction in the energy consumed to train FFNs using the proposed phantom parallel approach when compared with conventional tensor parallel methods. Additionally, the proposed approach is shown to train smaller phantom models to the same model loss on smaller GPU counts as larger tensor parallel models on larger GPU counts offering the possibility for even greater energy savings.

Problem

• Reducing energy consumption in large neural network training
• Improving efficiency of tensor parallelism in model training
• Enhancing communication efficiency in large model inferencing

Innovation

• Phantom parallelism reduces energy consumption
• Custom autograd operations for efficient training
• Lower bandwidth and FLOP counts demonstrated