🤖 AI Summary
This work addresses the high latency, substantial memory footprint, and numerical instability—such as NaN outputs under FP16 precision—that hinder real-time inference of Transformer models. Building upon NVIDIA TensorRT, the authors develop a modular, containerized GPU-accelerated inference pipeline featuring a structure-aware mixed-precision strategy: FP16 is employed in linear layers while Softmax and LayerNorm operations retain FP32 to prevent overflow and preserve output fidelity, achieving cosine similarity ≥0.9998. A reproducible evaluation framework spanning over 360 configurations reveals that random inputs severely underestimate FP16 instability. Experiments demonstrate up to a 64.4× speedup over CPU baselines (single-sample latency <10 ms), a 63% reduction in memory usage, no degradation in downstream task accuracy, and stable FP16 acceleration ratios between 1.84× and 2.00×.
📝 Abstract
This paper presents the design and evaluation of a GPU-accelerated inference pipeline for transformer models using NVIDIA TensorRT with mixed-precision optimization. We evaluate BERT-base (110M parameters) and GPT-2 (124M parameters) across batch sizes from 1 to 32 and sequence lengths from 32 to 512. The system achieves up to 64.4x speedup over CPU baselines, sub-10 ms latency for single-sample inference, and a 63 percent reduction in memory usage. We introduce a hybrid precision strategy that preserves FP32 for numerically sensitive operations such as softmax and layer normalization, while applying FP16 to linear layers. This approach maintains high numerical fidelity (cosine similarity >= 0.9998 relative to baseline outputs) and eliminates NaN instability. The pipeline is implemented as a modular, containerized system that enables reproducible benchmarking across more than 360 configurations. Cross-GPU validation on an NVIDIA A100 shows consistent FP16 speedup ratios between 1.84x and 2.00x, along with stable numerical behavior. Downstream evaluation on SST-2 demonstrates no accuracy degradation under hybrid precision. Validation on WikiText-2 shows that random inputs underestimate NaN instability by up to 6x for full FP16, while confirming the robustness of the hybrid approach (0.0 percent NaN, cosine similarity >= 0.9998). These results provide a detailed characterization of performance and accuracy trade-offs across GPU architectures and offer practical guidance for deploying transformer models in latency-critical environments.