An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures

2025-01-14

AI Summary

Orthogonal convolutions suffer from computational inefficiency and incompatibility with modern convolutional operators—such as strided, dilated, grouped, and transposed convolutions—in large-scale CNNs, limiting their modeling capacity and practical applicability. To address this, we propose Adaptive Orthogonal Convolutions (AOC), the first scalable, fully functional orthogonal convolution framework. AOC achieves strict orthogonality via parameterized orthogonal matrix optimization coupled with structured kernel decomposition, enabling seamless integration of all standard convolution variants. Complemented by gradient-stabilizing training strategies and a modular library architecture, AOC significantly improves training efficiency and numerical robustness. Extensive experiments demonstrate that AOC delivers superior representational power and near-linear speedup across image classification and generative tasks—with performance gains scaling favorably with model size. The implementation is publicly released as the Orthogonium library.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Non-convex Optimization
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Orthogonal convolutional layers are the workhorse of multiple areas in machine learning, such as adversarial robustness, normalizing flows, GANs, and Lipschitzconstrained models. Their ability to preserve norms and ensure stable gradient propagation makes them valuable for a large range of problems. Despite their promise, the deployment of orthogonal convolution in large-scale applications is a significant challenge due to computational overhead and limited support for modern features like strides, dilations, group convolutions, and transposed convolutions.In this paper, we introduce AOC (Adaptative Orthogonal Convolution), a scalable method for constructing orthogonal convolutions, effectively overcoming these limitations. This advancement unlocks the construction of architectures that were previously considered impractical. We demonstrate through our experiments that our method produces expressive models that become increasingly efficient as they scale. To foster further advancement, we provide an open-source library implementing this method, available at https://github.com/thib-s/orthogonium.

Problem

• Orthogonal Convolution Layers
• Computational Efficiency
• Functional Limitations

Innovation

• Adaptive Orthogonal Convolution
• Efficiency Improvement
• Public Toolkit