Inheritance Between Feedforward and Convolutional Networks via Model Projection

2026-02-05

AI Summary

This work establishes that feedforward networks constitute a strict subset of generalized convolutional networks and identifies a parameterization mismatch between the two architectures. To address this, the authors propose a model projection method based on a unified tensor-activation formulation: by freezing pretrained convolutional filters and learning only scalar gating parameters, the approach enables parameter-efficient transfer learning. This strategy effectively inherits optimization techniques developed for feedforward networks while drastically reducing the number of trainable parameters. Using only simple training protocols, the method achieves strong transfer performance across multiple ImageNet-pretrained backbones and downstream image classification tasks.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Techniques for feedforward networks (FFNs) and convolutional networks (CNNs) are frequently reused across families, but the relationship between the underlying model classes is rarely made explicit. We introduce a unified node-level formalization with tensor-valued activations and show that generalized feedforward networks form a strict subset of generalized convolutional networks. Motivated by the mismatch in per-input parameterization between the two families, we propose model projection, a parameter-efficient transfer learning method for CNNs that freezes pretrained per-input-channel filters and learns a single scalar gate for each (output channel, input channel) contribution. Projection keeps all convolutional layers adaptable to downstream tasks while substantially reducing the number of trained parameters in convolutional layers. We prove that projected nodes take the generalized FFN form, enabling projected CNNs to inherit feedforward techniques that do not rely on homogeneous layer inputs. Experiments across multiple ImageNet-pretrained backbones and several downstream image classification datasets show that model projection is a strong transfer learning baseline under simple training recipes.

Problem

• feedforward networks
• convolutional networks
• model projection
• transfer learning
• parameter efficiency

Innovation

• model projection
• parameter-efficient transfer learning
• generalized feedforward networks
• convolutional networks
• tensor-valued activations