Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven

2026-05-07

AI Summary

This work addresses the limitation of the Randomized Hadamard Transform (RHT) in replicating the distributional properties of a Uniform Random Rotation (URR), particularly under worst-case scenarios, which hinders its quantization performance. To overcome this, the authors propose stacking multiple RHT layers and theoretically demonstrate that two layers suffice to approximate the marginal distribution of any single coordinate under URR, while three layers achieve the weak correlation structure required for effective vector quantization. A linear-time dynamic adaptation strategy is also introduced. Experimental results on compression frameworks such as DRIVE and QUIC-FL show that two-layer RHT attains asymptotically equivalent performance to URR, and with three layers, the vector quantization error rapidly converges to the URR baseline as dimensionality increases.

Abstract

Uniform random rotations (URRs) are a common preprocessing step in modern quantization approaches used for gradient compression, inference acceleration, KV-cache compression, model weight quantization, and approximate nearest-neighbor search in vector databases. In practice, URRs are often replaced by randomized Hadamard transforms (RHTs), which preserve orthogonality while admitting fast implementations. The remaining issue is the performance for worst-case inputs. With a URR, each coordinate is individually distributed as a shifted beta distribution, which converges to a Gaussian distribution in high dimensions. Generally, one RHT is not suitable in the worst case, as individual coordinates can be far from these distributions. We show that after composing two RHTs on any $d$-sized input vector, the marginal distribution of every fixed coordinate of the normalized rotated vector is within $O(d^{-1/2})$ of a standard Gaussian both in Kolmogorov distance and in $1$-Wasserstein distance. We then plug these bounds into the analyses of modern compression schemes, namely DRIVE and QUIC-FL, and show that two RHTs achieve performance that asymptotically matches URRs. However, we show that two RHTs may not be sufficient for Vector Quantization (VQ), which often requires weak correlation across fixed-size blocks of coordinates (as opposed to only marginal distribution convergence for single coordinates). We prove that a composition of three RHTs leads to decaying coordinate covariance. This ensures that any fixed, bounded, multi-dimensional VQ codebook optimized for URRs has the same expected error when using three RHTs, up to an additive term that vanishes with the dimension. Finally, because practical inputs are rarely adversarial, we propose a linear-time ${O}(d)$ check on the input's moments to dynamically adapt the number of RHTs used at runtime to improve performance.

Problem

• quantization
• randomized Hadamard transforms
• uniform random rotations
• vector quantization
• worst-case inputs

Innovation

• Randomized Hadamard Transforms
• Uniform Random Rotations
• Vector Quantization
• Gradient Compression
• Gaussian Approximation