pytorch

This work addresses the lack of transparent, scalable, and deeply PyTorch-integrated open-source tools for post-training large language models, which hinders research iteration and deployment efficiency. We propose a native PyTorch-based, modular post-training framework centered on the principle of “hackability,” offering composable model builders, training recipes, and a distributed training stack that support diverse fine-tuning strategies and hardware configurations. While maintaining high performance and memory efficiency, the framework significantly enhances code transparency and research flexibility. Empirical evaluations demonstrate that it matches or even surpasses mainstream tools such as Axolotl and Unsloth across multiple post-training scenarios, thereby facilitating efficient and reproducible scientific exploration.

To address inefficient backpropagation and excessive memory overhead in small-scale computational graph training on single-CPU nodes, this paper proposes BurTorch, a lightweight training framework. Methodologically, BurTorch abandons the abstraction-layer bloat of general-purpose deep learning frameworks and instead embraces the fundamentals of compiled-language programming and numerical optimization: it employs hand-optimized C++, static computational graphs, explicit memory management, and a native backpropagation engine grounded in the Linnainmaa/Rumelhart chain rule. This system-level design directly targets performance bottlenecks inherent to small-graph training. Experiments demonstrate that, on representative small-graph tasks, BurTorch achieves up to 2000× speedup and 3500× memory reduction over PyTorch; even on a micro-scale GPT-3 model, it delivers 20× acceleration and 80× memory savings. These results significantly advance efficient, resource-constrained deep learning training.

Existing LLM pretraining frameworks suffer from fragmentation, poor interoperability, and high maintenance overhead, severely hindering systematic evaluation and production deployment of training methodologies. This paper introduces an open-source, PyTorch-native distributed training system designed for models ranging from 10B to 400B parameters. It proposes a novel modular 3D parallelism architecture—integrating data, tensor, and pipeline parallelism—and tightly couples Float8 quantization with SymmetricMemory hardware-software co-design. The system further incorporates elastic scaling, unified checkpointing, and a reproducible experiment platform. Evaluated on the Llama 3.1 series, it achieves 65.08% speedup on 128 GPUs (8B model), an additional 12.59% gain on 256 GPUs (70B), and a further 30% improvement on 512 GPUs (405B), significantly outperforming baseline systems. The framework delivers high performance, strong scalability, and production readiness.

Existing GFlowNet research lacks a unified, scalable PyTorch implementation framework, hindering the development of novel training objectives, integration with diverse environments, and reproducible benchmarking. To address this, we introduce the first modular, object-oriented open-source GFlowNet library built on PyTorch. Our method standardizes environment interfaces and sampler abstractions, enables plug-and-play loss functions—including trajectory balance (TB), detailed balance (DB), and unnormalized balance (UB)—and decouples state-space representation, action policies, and flow parameterizations to facilitate customization and composability. The framework successfully reproduces multiple state-of-the-art results across canonical benchmarks, substantially lowering the barrier for algorithm validation and extension. The codebase is publicly released and has been widely adopted by the research community.

Hand-tuned deep learning optimizations (e.g., FlashAttention) suffer from low efficiency and poor interpretability, while existing automatic compilation approaches lack generalizability and hardware awareness. Method: We propose an I/O-aware Neural Circuit Graph (NCG) optimization framework that explicitly models GPU microarchitecture, resource constraints, and memory hierarchy as a relabelable graph structure—enabling joint derivation of streaming computation, tiling schedules, and higher-order effects (e.g., quantization, multi-level storage). It establishes a formal bridge from hardware behavioral assumptions to performance assertions and supports graphical modeling of IR-level pseudocode alongside GPU-driven performance modeling. Contribution/Results: NCG is the first framework to systematically reverse-engineer the intrinsic mechanisms of state-of-the-art techniques like FlashAttention. It establishes a GPU algorithm design paradigm that is interpretable, evolvable, and formally verifiable—outperforming both manual tuning and black-box auto-compilation by significant margins.

Industrial-scale recommendation and ranking models feature highly complex and continuously evolving architectures, rendering traditional optimization approaches—based on manual intervention or module-level rules—difficult to scale. This work proposes the first extensible and customizable operator-level automatic transformation framework integrated into PyTorch 2.x. By leveraging FX intermediate representation, the PT2 compiler, predefined pattern matching, and a greedy search algorithm, the framework achieves general-purpose model optimizations while strictly preserving computational semantics. Evaluated on real-world industrial recommendation models, the approach delivers up to 63% inference speedup, a 6% reduction in peak memory usage, and over 400 seconds of compilation time savings. The implementation has been open-sourced as part of PyTorch 2.x.

This work proposes a training-free method for constructing neural networks that precisely simulate arbitrary Turing machines. Given a formal description of a Turing machine, the system automatically compiles it into a corresponding neural network whose forward pass exactly mirrors a single step of the machine’s execution. Grounded in first principles, the approach employs ReLU networks to implement Boolean logic and adders, leverages Cantor set encoding together with hard attention mechanisms to manage tape read–write operations, and demonstrates Turing completeness within both Transformer architectures—incorporating self-attention and cross-attention—and recurrent neural network frameworks. This study establishes a reproducible and verifiable theoretical and practical foundation for integrating neural and symbolic computation.

Manually crafting high-performance GPU kernels is inefficient and struggles to meet the computational demands of modern machine learning. This work proposes the first end-to-end verifiable autonomous agent framework that automatically analyzes bottlenecks in PyTorch models—guided by Amdahl’s Law—and iteratively generates and optimizes Triton or CUDA C++ kernels without human intervention, employing a five-stage correctness verification mechanism. The approach comprehensively covers core operators in mainstream Transformer architectures and achieves substantial speedups over both PyTorch eager execution and torch.compile (max-autotune) on H100 GPUs—for instance, accelerating RMSNorm by 5.29× and softmax by 2.82×—while attaining top performance on the B200 vectorsum_v2 benchmark.

This work addresses the bottleneck in large-scale deep learning training, which often stems from system implementation rather than model architecture. To this end, the authors develop a lightweight, general-purpose training framework natively implemented in C++ and CUDA from first principles, integrating tensor computation, reverse-mode automatic differentiation, an efficient caching allocator, multi-modal distributed execution, and an MLIR-based compiler. This design achieves fine-grained hardware control while preserving modeling simplicity. On an 8-GPU RTX 6000 Ada system, the framework trains a 124-million-parameter GPT-2 model at 407K tokens/s—surpassing PyTorch’s 395K tokens/s—while reducing memory consumption by 22% and achieving a lower validation loss. This represents the first fully native, end-to-end tunable high-performance training system built solely on standard C++ and core CUDA primitives.

This work addresses the challenge of defect detection in deep learning libraries such as TensorFlow and PyTorch, where complex APIs often lead to subtle bugs and existing testing approaches suffer from high false-positive rates due to imprecise specifications. To overcome this limitation, the authors propose a machine learning classifier that leverages tensor shape abstraction as a precise input representation for API validity constraints. By integrating runtime feedback to automatically generate labeled training data, the method learns accurate usage patterns without relying on manual annotations. Implemented within the ACETest framework, the approach achieves over 91% classification accuracy across 183 APIs and significantly improves test pass rates—from 29% to 61%—demonstrating enhanced precision and scalability in testing deep learning libraries.