xla

This work addresses the lack of efficient, differentiable, and multi-GPU-scalable native sparse matrix operations in PyTorch for industrial-scale scientific computing. The authors propose an open-source PyTorch library that enables automatic differentiation via the adjoint method, achieving O(1) computational graph nodes and O(nnz) memory footprint. The library integrates Newton, Picard, and Anderson nonlinear solvers alongside eigenvalue computation routines. Leveraging domain decomposition and halo exchange, it supports multi-GPU scaling and successfully solves sparse linear systems with 400 million degrees of freedom across three GPUs. By seamlessly interfacing with PyTorch’s autograd system, the library drastically reduces memory overhead and provides efficient, native support for large-scale end-to-end differentiable scientific simulations.

Existing LLM pretraining frameworks suffer from fragmentation, poor interoperability, and high maintenance overhead, severely hindering systematic evaluation and production deployment of training methodologies. This paper introduces an open-source, PyTorch-native distributed training system designed for models ranging from 10B to 400B parameters. It proposes a novel modular 3D parallelism architecture—integrating data, tensor, and pipeline parallelism—and tightly couples Float8 quantization with SymmetricMemory hardware-software co-design. The system further incorporates elastic scaling, unified checkpointing, and a reproducible experiment platform. Evaluated on the Llama 3.1 series, it achieves 65.08% speedup on 128 GPUs (8B model), an additional 12.59% gain on 256 GPUs (70B), and a further 30% improvement on 512 GPUs (405B), significantly outperforming baseline systems. The framework delivers high performance, strong scalability, and production readiness.

This paper addresses the challenge of automatically mapping PyTorch models to synthesizable hardware. We propose an open-source, end-to-end compilation toolchain that innovatively integrates Allo (an accelerator design language), Calyx (a hardware intermediate representation), and CIRCT (an LLVM-based hardware compilation framework). A key methodological contribution is a memory-partitioning compilation pass tailored for memory-intensive machine learning workloads, which significantly enhances data parallelism and on-chip memory efficiency. Our core contributions are threefold: (1) the first fully automated, synthesizable translation from PyTorch frontend to SystemVerilog RTL; (2) a memory optimization strategy that preserves functional correctness while substantially reducing off-chip bandwidth pressure; and (3) experimental validation showing that the generated FPGA implementations achieve throughput and resource utilization comparable to industrial-grade, closed-source tools such as Vitis HLS.

In PyTorch 2, FX graph breaks—caused primarily by dynamic control flow and unsupported Python I/O operations—fragment computational graphs, trigger frequent fallbacks to eager mode, exacerbate CPU-GPU synchronization overhead, and hinder graph-level optimizations. Method: We propose a compiler frontend source-to-source transformation technique that automatically analyzes and rewrites Python code prior to TorchDynamo’s graph capture. Leveraging the Jac framework, our approach integrates static analysis with targeted source rewriting to eliminate *repairable* graph breaks, enabling seamless interoperability between TorchDynamo and TorchInductor. Contribution/Results: Evaluated on eight Hugging Face models, our method achieves zero graph breaks for six models, reduces inference latency by up to 75%, and improves end-to-end throughput by up to 8%. This work is the first to systematically incorporate source-level transformations into the PyTorch JIT compilation pipeline, significantly expanding both the scope of compilable graphs and the applicability of downstream optimizations.

This work addresses the lack of transparent, scalable, and deeply PyTorch-integrated open-source tools for post-training large language models, which hinders research iteration and deployment efficiency. We propose a native PyTorch-based, modular post-training framework centered on the principle of “hackability,” offering composable model builders, training recipes, and a distributed training stack that support diverse fine-tuning strategies and hardware configurations. While maintaining high performance and memory efficiency, the framework significantly enhances code transparency and research flexibility. Empirical evaluations demonstrate that it matches or even surpasses mainstream tools such as Axolotl and Unsloth across multiple post-training scenarios, thereby facilitating efficient and reproducible scientific exploration.

This work addresses the performance bottleneck in modern deep learning caused by the overhead of frequent GPU kernel launches due to numerous small-scale tensor operations. To mitigate this, the authors propose a persistent GPU kernel runtime system that continuously schedules computational tasks via a host-side task queue and dynamically loads operators through NVRTC-based just-in-time compilation and function pointer injection, thereby eliminating redundant kernel launches. The design introduces an innovative dual-slot aliasing mechanism for concurrent updates and a generic tensor abstraction, enabling transparent integration with PyTorch via TorchDispatch. Evaluated on representative workloads dominated by small operations, the system achieves up to 15.3× speedup over standard PyTorch, substantially improves GPU utilization, and maintains strong ecosystem compatibility.

This work addresses the challenge of efficiently generating vector-length-agnostic (VLA) machine learning code for scalable vector instruction sets such as Arm SVE, where unknown vector lengths at compile time hinder traditional compilers. The authors present the first end-to-end VLA support in MLIR/IREE, introducing a vector-length-aware compact data layout and unifying dynamic tiling, operator fusion, and scalable vectorization within a single compilation framework. Evaluated on Arm CPUs, the generated SVE code achieves up to 1.45× speedup over IREE’s NEON implementation, outperforms multiple frameworks in the PyTorch ecosystem, and demonstrates strong scalability with increasing vector lengths in simulation, effectively balancing performance and hardware portability.

Existing scientific computing codes are difficult to efficiently port to specialized architectures such as AMD AI Engine, often requiring extensive manual refactoring. This work proposes a tensor abstraction–based compilation approach that automatically elevates generic loops to tensor semantics by parsing lightweight OpenMP annotations, and constructs an end-to-end compilation pipeline to map computations onto the AI Engine execution model. The method significantly reduces programming complexity through minimal OpenMP directives and enables CPU–NPU cooperative scheduling. Experimental results on six scientific and AI kernel benchmarks show that the NPU achieves higher energy efficiency than a multi-core CPU at float32 precision; for two kernels, cooperative execution yields a 40% performance improvement and 15% energy reduction.

Industrial-scale recommendation and ranking models feature highly complex and continuously evolving architectures, rendering traditional optimization approaches—based on manual intervention or module-level rules—difficult to scale. This work proposes the first extensible and customizable operator-level automatic transformation framework integrated into PyTorch 2.x. By leveraging FX intermediate representation, the PT2 compiler, predefined pattern matching, and a greedy search algorithm, the framework achieves general-purpose model optimizations while strictly preserving computational semantics. Evaluated on real-world industrial recommendation models, the approach delivers up to 63% inference speedup, a 6% reduction in peak memory usage, and over 400 seconds of compilation time savings. The implementation has been open-sourced as part of PyTorch 2.x.