VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination

2026-02-11

arXiv.org

AI Summary

This work addresses the limited optimization of data movement in existing DNN compilers, which typically target only a subset of operators and struggle with redundant transfers in large models. To overcome this, the authors propose VTC, a novel framework that introduces the concept of virtual tensors, replacing physical data movement with index mapping and integrating a new data-movement elimination algorithm to uniformly optimize across the full spectrum of data-movement operators. VTC enables seamless optimization for arbitrary tensor operator compositions and interoperates effectively with existing compute kernels. Evaluated on NVIDIA GPUs, VTC achieves an average speedup of 1.28× (up to 1.93×) and reduces inference memory consumption by 17.5% on average (up to 60%).

Abstract

With the widening gap between compute and memory operation latencies, data movement optimizations have become increasingly important for DNN compilation. Current optimizations such as layout transformations and operator fusion only target a subset of tensor operators and consequently miss important opportunities for reducing data movement in contemporary DNN workloads, including large language models. We introduce VTC, a novel tensor compilation framework that for the first time eliminates all unnecessary data movement by targeting the full spectrum of data movement operators. VTC proposes the concept of virtual tensors to track data movement between compute operators via index mappings rather than expensive physical data transfers to and from global memory, which can seamlessly interoperate with existing computation kernels and handle arbitrary tensor operator compositions. We also introduce a novel data movement elimination algorithm to automatically identify a profitable virtual tensor creation strategy. Evaluation on a variety of DNNs shows that VTC can outperform existing ML compilers by up to 1.93x (1.28x on average) on NVIDIA GPUs with up to 60% (17.5% on average) inference memory savings.

Problem

• data movement elimination
• DNN compilation
• tensor operators
• memory optimization
• virtual tensors

Innovation

• virtual tensors
• data movement elimination
• DNN compilation
• index mapping
• tensor operator fusion