DEEP-GAP: Deep-learning Evaluation of Execution Parallelism in GPU Architectural Performance

2026-04-15

AI Summary

This study addresses the lack of systematic quantification of inference performance differences between NVIDIA T4 and L4 GPUs under controlled conditions. Following the GDEV-AI methodology, we evaluate inference throughput and energy efficiency of ResNet-family models across FP32, FP16, and INT8 precisions using PyTorch and TensorRT on identically configured systems. Our analysis reveals, for the first time, that the L4 significantly outperforms the T4 in low-latency scenarios with small batch sizes (e.g., batch=1–8), whereas the T4 remains competitive at larger batch sizes. INT8 inference achieves up to 58× higher throughput than a CPU baseline, and the L4 delivers up to 4.4× greater throughput than the T4, attaining peak energy efficiency at batch sizes of 16–32—effectively balancing latency and throughput.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Modern datacenters increasingly rely on low-power, single-slot inference accelerators to balance performance, energy efficiency, and rack density constraints. The NVIDIA T4 GPU has become widely deployed due to strong performance per watt and mature software support. Its successor, the NVIDIA L4 GPU, introduces improvements in Tensor Core throughput, cache capacity, memory bandwidth, and parallel execution capability. However, limited empirical evidence quantifies the practical inference performance gap between these two generations under controlled and reproducible conditions. This work introduces DEEP-GAP, a systematic evaluation extending the GDEV-AI methodology to GPU inference. Using identical configurations and workloads, we evaluate ResNet18, ResNet50, and ResNet101 across FP32, FP16, and INT8 precision modes using PyTorch and TensorRT. Results show that reduced precision significantly improves performance, with INT8 achieving up to 58x throughput improvement over CPU baselines. L4 achieves up to 4.4x higher throughput than T4 while reaching peak efficiency at smaller batch sizes between 16 and 32, improving latency-throughput tradeoffs for latency-sensitive workloads. T4 remains competitive for large batch workloads where cost or power efficiency is important. DEEP-GAP provides practical guidance for selecting precision modes, batch sizes, and GPU architectures for modern inference deployments.

Problem

• GPU inference
• performance evaluation
• execution parallelism
• precision modes
• throughput

Innovation

• DEEP-GAP
• GPU inference evaluation
• execution parallelism
• precision modes
• throughput-latency tradeoff