Decorrelation Speeds Up Vision Transformers
Vision Transformers (ViTs) pretrained via Masked Autoencoders (MAEs) achieve strong performance under low-label regimes, yet their high computational cost hinders industrial deployment. To address this, we propose **Selective Decorrelation Backpropagation (DecorrBP)**—a lightweight optimization technique that imposes layer-wise gradient covariance constraints exclusively within the MAE encoder, enhancing gradient propagation efficiency and convergence speed while preserving training stability. Evaluated on ImageNet-1K, DecorrBP reduces pretraining time by 21.1% and carbon emissions by 21.4%. On downstream ADE20K semantic segmentation, it improves mIoU by 1.1 points; consistent gains are also observed on industrial datasets. Crucially, DecorrBP is the first method to integrate gradient decorrelation into the MAE training framework without modifying model architecture or loss functions—enabling efficient, low-carbon, and high-performance ViT pretraining.