Deep residual learning with product units

2025-05-07

AI Summary

Balancing expressive power, parameter efficiency, structural fidelity, and noise robustness remains challenging for deep convolutional networks. To address this, we propose PURe—a novel architecture that systematically embeds 2D multiplicative units in the second layer of residual blocks, replacing conventional convolution followed by nonlinear activation to enable high-order feature interaction modeling. Crucially, PURe eliminates explicit activation functions, thereby preserving input structural information and enhancing gradient propagation stability. The model is trained end-to-end under supervised learning without auxiliary regularization. Extensive experiments demonstrate consistent superiority over same-depth ResNets across Galaxy10, CIFAR-10, and ImageNet. Specifically, PURe-34 achieves 80.27% top-1 accuracy on ImageNet—surpassing ResNet-34—while reducing parameters by 40%, accelerating convergence fivefold, and exhibiting significantly improved robustness to Poisson noise.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Non-convex Optimization

Application Category

• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We propose a deep product-unit residual neural network (PURe) that integrates product units into residual blocks to improve the expressiveness and parameter efficiency of deep convolutional networks. Unlike standard summation neurons, product units enable multiplicative feature interactions, potentially offering a more powerful representation of complex patterns. PURe replaces conventional convolutional layers with 2D product units in the second layer of each residual block, eliminating nonlinear activation functions to preserve structural information. We validate PURe on three benchmark datasets. On Galaxy10 DECaLS, PURe34 achieves the highest test accuracy of 84.89%, surpassing the much deeper ResNet152, while converging nearly five times faster and demonstrating strong robustness to Poisson noise. On ImageNet, PURe architectures outperform standard ResNet models at similar depths, with PURe34 achieving a top-1 accuracy of 80.27% and top-5 accuracy of 95.78%, surpassing deeper ResNet variants (ResNet50, ResNet101) while utilizing significantly fewer parameters and computational resources. On CIFAR-10, PURe consistently outperforms ResNet variants across varying depths, with PURe272 reaching 95.01% test accuracy, comparable to ResNet1001 but at less than half the model size. These results demonstrate that PURe achieves a favorable balance between accuracy, efficiency, and robustness. Compared to traditional residual networks, PURe not only achieves competitive classification performance with faster convergence and fewer parameters, but also demonstrates greater robustness to noise. Its effectiveness across diverse datasets highlights the potential of product-unit-based architectures for scalable and reliable deep learning in computer vision.

Problem

• Enhancing deep network expressiveness with product units
• Improving parameter efficiency in convolutional networks
• Achieving robustness and faster convergence in classification

Innovation

• Integrates product units into residual blocks
• Replaces convolutional layers with 2D product units
• Eliminates nonlinear activation functions for structure