Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

2026-08-13

AI Summary

This work addresses the substantial computational overhead in large language model inference caused by repetitive high-dimensional matrix multiplications. The authors propose a training-free, input-adaptive acceleration method that dynamically selects informative slices along the contraction dimension to reduce matrix multiplication operations in Transformers, without altering model weights. They reveal an intrinsic structural asymmetry within Transformers—attention modules are significantly more compressible than MLP blocks—and leverage this insight to design an input-adaptive slice selection strategy, a retention rate control mechanism, and a custom CUDA kernel. Evaluated across models ranging from 1B to 70B parameters, the approach achieves notable reductions in computation and real-world speedups on long-sequence tasks, demonstrating broad applicability in cross-modal and multi-task settings.

Abstract

Transformer-based language models achieve strong performance but incur substantial inference cost due to repeated high-dimensional matrix multiplications. We propose Reduced Matrix Multiplication (RMM), a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along their contraction dimensions, without modifying model weights. Under a simple retention-ratio control, RMM provides a smooth and predictable accuracy-efficiency trade-off. Across language models ranging from 1B to 70B parameters, we find that reduction tolerance depends on the model family, task, component, and retention ratio, although it often improves with model scale. Under moderate reduction, RMM remains robust across the evaluated discriminative, autoregressive generation, and long-context settings. We further show that the same principle extends to multimodal vision-language inference. Mechanistic ablations reveal a structural asymmetry within Transformers: attention-side computations are substantially more reducible than MLP components. Finally, wall-clock benchmarks with custom kernels on an NVIDIA A100 show that these computational savings can translate into practical runtime gains, especially at longer sequence lengths. Together, these results position RMM as a scalable direction for input-adaptive inference-time optimization.

Problem

• matrix multiplication
• LLM inference
• computational efficiency
• input-adaptive
• Transformer

Innovation

• Reduced Matrix Multiplication
• input-adaptive inference
• Transformer acceleration
• matrix-product reduction
• training-free optimization