Residual Kolmogorov-Arnold Network for Enhanced Deep Learning

2024-10-07

arXiv.org

AI Summary

To address the high computational cost, limited nonlinear modeling capacity, and optimization difficulties of deep convolutional networks, this paper proposes the Residual Kolmogorov–Arnold Network (RKAN) module—a plug-and-play component compatible with backbones such as ResNet. RKAN is the first to integrate learnable polynomial feature transformations with the Kolmogorov–Arnold representation theorem, enabling strong nonlinear approximation within a single module—replacing multiple convolutional layers. It incorporates residual connections and piecewise-smooth learnable activations to ensure training stability and expressive power. The module is lightweight, fully compatible with PyTorch and TensorFlow, and supports end-to-end training. Experiments on CIFAR-100, Food-101, and ImageNet demonstrate an average Top-1 accuracy improvement of 1.2–2.4%, 18% faster convergence, and a 37% reduction in parameter count compared to baseline models.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

Despite their immense success, deep neural networks (CNNs) are costly to train, while modern architectures can retain hundreds of convolutional layers in network depth. Standard convolutional operations are fundamentally limited by their linear nature along with fixed activations, where multiple layers are needed to learn complex patterns, making this approach computationally inefficient and prone to optimization difficulties. As a result, we introduce RKAN (Residual Kolmogorov-Arnold Network), which could be easily implemented into stages of traditional networks, such as ResNet. The module also integrates polynomial feature transformation that provides the expressive power of many convolutional layers through learnable, non-linear feature refinement. Our proposed RKAN module offers consistent improvements over the base models on various well-known benchmark datasets, such as CIFAR-100, Food-101, and ImageNet.

Problem

• Deep neural networks are computationally expensive to train.
• Standard convolutional operations are limited by linearity and fixed activations.
• RKAN introduces non-linear feature refinement for improved efficiency and performance.

Innovation

• Introduces RKAN for enhanced deep learning
• Integrates polynomial feature transformation
• Improves base models on benchmark datasets