megatron-lm

This work addresses the high latency, substantial memory footprint, and numerical instability—such as NaN outputs under FP16 precision—that hinder real-time inference of Transformer models. Building upon NVIDIA TensorRT, the authors develop a modular, containerized GPU-accelerated inference pipeline featuring a structure-aware mixed-precision strategy: FP16 is employed in linear layers while Softmax and LayerNorm operations retain FP32 to prevent overflow and preserve output fidelity, achieving cosine similarity ≥0.9998. A reproducible evaluation framework spanning over 360 configurations reveals that random inputs severely underestimate FP16 instability. Experiments demonstrate up to a 64.4× speedup over CPU baselines (single-sample latency <10 ms), a 63% reduction in memory usage, no degradation in downstream task accuracy, and stable FP16 acceleration ratios between 1.84× and 2.00×.

Existing LLM pretraining frameworks suffer from fragmentation, poor interoperability, and high maintenance overhead, severely hindering systematic evaluation and production deployment of training methodologies. This paper introduces an open-source, PyTorch-native distributed training system designed for models ranging from 10B to 400B parameters. It proposes a novel modular 3D parallelism architecture—integrating data, tensor, and pipeline parallelism—and tightly couples Float8 quantization with SymmetricMemory hardware-software co-design. The system further incorporates elastic scaling, unified checkpointing, and a reproducible experiment platform. Evaluated on the Llama 3.1 series, it achieves 65.08% speedup on 128 GPUs (8B model), an additional 12.59% gain on 256 GPUs (70B), and a further 30% improvement on 512 GPUs (405B), significantly outperforming baseline systems. The framework delivers high performance, strong scalability, and production readiness.

To address parallelization bottlenecks hindering training efficiency of large-scale Mixture-of-Experts (MoE) models across thousands of GPUs, this work proposes a five-dimensional heterogeneous hybrid parallelism framework. It introduces MoE Parallel Folding—a novel mechanism that decouples parallelization strategies for attention and MoE layers, enabling their independent, optimal configuration. A dynamic token-level dispatcher is designed to support both token-dropping and dropless routing. The framework unifies tensor, expert, sequence-context, data, and pipeline parallelism, and—built upon Megatron-Core—supports dynamic tensor shapes and cross-dimensional coordinated scheduling. Evaluated on an H100 cluster, the framework achieves 49.3% MFU for Mixtral 8×22B and 39.0% MFU for Qwen2-57B-A14B, demonstrating strong scalability up to 1,000 GPUs and native support for sequences up to 128K tokens.

This work addresses the communication bottleneck caused by frequent inter-device synchronization in multi-GPU tensor-parallel inference, which limits scalability. The authors propose the Parallel Track Transformer architecture, featuring an innovative parallel-track computation structure and a novel task partitioning and scheduling strategy that substantially reduces cross-device dependencies while preserving model quality. Implemented within the TensorRT-LLM and vLLM frameworks, the approach achieves up to a 16-fold reduction in synchronization operations, decreases first-token latency by 15–30%, reduces per-token generation time by 2–12%, and improves throughput by as much as 31.9%.

To address the high GPU memory consumption and hardware requirements in training large language models (LLMs) with ultra-long contexts, this paper proposes a fully pipelined distributed Transformer architecture. Its core innovation is a novel sequence-chunking pipeline mechanism that extends maximum sequence length by 16× without modifying the model architecture, while maintaining full compatibility with existing training techniques—including pipeline parallelism, FP16/BF16 mixed-precision training, and computational graph reordering. We successfully train an 8B-parameter model on just four GPUs to handle sequences up to 2 million tokens, achieving a sustained model FLOPs utilization (MFU) of over 55%, significantly outperforming state-of-the-art approaches. This method substantially reduces hardware dependency and resource costs for long-context LLM training, establishing a general, scalable, and efficient paradigm for ultra-long-context LLM training.

This work addresses the lack of transparent, scalable, and deeply PyTorch-integrated open-source tools for post-training large language models, which hinders research iteration and deployment efficiency. We propose a native PyTorch-based, modular post-training framework centered on the principle of “hackability,” offering composable model builders, training recipes, and a distributed training stack that support diverse fine-tuning strategies and hardware configurations. While maintaining high performance and memory efficiency, the framework significantly enhances code transparency and research flexibility. Empirical evaluations demonstrate that it matches or even surpasses mainstream tools such as Axolotl and Unsloth across multiple post-training scenarios, thereby facilitating efficient and reproducible scientific exploration.

This work addresses the computational scalability and training stability challenges inherent in pretraining extremely large-scale Mixture-of-Experts (MoE) language models. Leveraging the Aurora supercomputer and a self-developed Optimus training framework, the authors efficiently pretrained the Mula series of MoE models from scratch, spanning from billions to 220 billion parameters. Key innovations include an EP-Aware sharded optimizer, custom GPU expert computation kernels, and a highly reliable distributed fault-tolerance mechanism. The system achieves approximately 90% strong scaling efficiency across 12,288 GPUs, yielding up to a 1.71× speedup in training throughput while successfully completing trillion-token-scale pretraining with high stability.

This work addresses the challenge of mismatched computational demands between vision encoders and large language models (LLMs) under long-context scenarios in multimodal large language model training, where conventional LLM-centric parallelization strategies fall short. The authors propose a heterogeneous parallel framework that decouples the encoder and LLM for the first time, enabling module-level independent parallelism and flexible device placement—either co-located or non-co-located. A boundary communicator ensures tensor semantic consistency across modules, while an extended scheduling mechanism supports diverse parallelism combinations, including tensor (TP), context (CP), pipeline (PP), data (DP), and expert (EP) parallelism. Experiments demonstrate that the proposed approach achieves up to 49.3% higher TFLOPS/GPU in co-located configurations and improves token throughput by 13.0% and TFLOPS/GPU by 9.6% in non-co-located settings, all while maintaining convergence performance on par with baseline methods.

This work addresses the performance bottleneck in Transformer training caused by non-GEMM operations—such as normalization, activation functions, and residual updates—that incur substantial global memory traffic. To mitigate this, the authors propose a GEMM-plus-epilogue programming abstraction that restructures non-attention computations in both forward and backward passes into fused patterns comprising a GEMM main loop followed by composable epilogue primitives. By executing these epilogue operations while GEMM outputs remain on-chip, the approach drastically reduces off-chip memory accesses. On GPUs, they design epilogue primitives supporting scaling, reduction, bivariate transformations, and accumulation, compatible with both hand-tuned and LLM-generated high-performance kernels. Evaluated across diverse Transformer workloads, the resulting CODA kernels achieve high throughput while maintaining framework usability and hardware efficiency.

This work addresses the GPU memory bottleneck in Transformer models caused by high parameter and activation memory demands during training and inference. The authors propose a novel parallelism strategy that integrates tensor parallelism (TP) and sequence parallelism (SP) along the same device axis, enabling each device to simultaneously shard both model weights and input sequences. By leveraging broadcast-based weight sharding with key-value exchange in attention layers and ring-based weight passing with local accumulation in gated MLPs, the method achieves dual compression of both parameter and activation memory. This approach significantly reduces per-device memory consumption, outperforming conventional TP, SP, and their hybrid variants. It demonstrates superior hardware adaptability and scaling efficiency under long-context and memory-constrained settings, while seamlessly integrating with pipeline and expert parallelism.