WBMM: Windowed Batch Matrix Multiplication for Efficient Large Receptive Field Convolution

2026-07-02

AI Summary

This work addresses the significant computational inefficiency of large-kernel depthwise convolutions, which stems from irregular gather-based memory access patterns that degrade performance as kernel size increases—existing acceleration methods often exacerbate this issue on large feature maps. To overcome this, the authors propose Windowed Batched Matrix Multiplication (WBMM), which partitions the input into contiguous windows and constructs weight matrices using a compact relative positional bias table, enabling efficient large-receptive-field convolution through regularized batched matrix multiplications. WBMM requires no specialized kernels, achieves consistent throughput gains across hardware platforms as window size grows—contrary to the performance degradation of conventional depthwise convolutions—and matches or exceeds baseline accuracy on ImageNet-1K, COCO, and ADE20K while accelerating training by 1.31–1.88×. Notably, a 14×14 WBMM configuration outperforms 5×5 depthwise convolution in speed while expanding the receptive field by 7.8×.

Abstract

Large kernel depthwise convolutions achieve strong performance but suffer from significant degradation as kernel size grows due to irregular memory access from gather-based computation; while Large Kernel Acceleration (LKA) helps on small feature maps, it becomes counterproductive on large feature maps, even slower than non-accelerated implementations. We propose Windowed Batch Matrix Multiplication (WBMM), which partitions input into contiguous windows and indexes a compact relative position bias table to construct weight matrices, enabling regular memory access via batched matrix multiplication. This yields a unique property: WBMM's throughput improves with larger windows, opposite to depthwise convolutions that degrade with larger kernels. Operator-level benchmarks show WBMM with 14x14 windows outperforms 5x5 depthwise convolution baselines in speed while providing a 7.8x larger per-layer receptive field. Combined with inter-block cross-window communication and hierarchical window reparameterization, WBMM achieves comparable or higher accuracy on ImageNet-1K, COCO, and ADE20K with 1.31-1.88x training speedup, and demonstrates consistent advantages across GPU, CPU, and edge devices without requiring specialized acceleration kernels. Our code is available at http://github.com/wansong-s/WBMM

Problem

• large kernel convolution
• memory access irregularity
• depthwise convolution
• computational efficiency
• receptive field

Innovation

• Windowed Batch Matrix Multiplication
• Large Receptive Field
• Depthwise Convolution
• Memory Access Optimization
• Efficient CNN