Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi

2025-01-22

AI Summary

This work investigates whether CNNs and Vision Transformers (ViTs) share unified learning mechanisms in image recognition, with a focus on the role of multi-head attention (MHA). To this end, we propose Single-Node Performance (SNP), a metric that quantifies the discriminative capability of individual feed-forward and MHA sub-module nodes toward label clusters, revealing common mechanisms of progressive signal enhancement and noise suppression. We further design the ANDC pruning method, achieving parameter-efficient compression without accuracy degradation. Notably, we discover— for the first time—spontaneous symmetry breaking among MHA heads, leading to head-wise label specialization, and establish a quantitative *modus vivendi* model characterizing their coexistence. Extensive experiments on CIFAR-100 and Flowers-102 validate the universality of these mechanisms, demonstrating both high accuracy and strong model compactness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Convolutional neural networks (CNNs) evaluate short-range correlations in input images which progress along the layers, whereas vision transformer (ViT) architectures evaluate long-range correlations, using repeated transformer encoders composed of fully connected layers. Both are designed to solve complex classification tasks but from different perspectives. This study demonstrates that CNNs and ViT architectures stem from a unified underlying learning mechanism, which quantitatively measures the single-nodal performance (SNP) of each node in feedforward (FF) and multi-head attention (MHA) subblocks. Each node identifies small clusters of possible output labels, with additional noise represented as labels outside these clusters. These features are progressively sharpened along the transformer encoders, enhancing the signal-to-noise ratio. This unified underlying learning mechanism leads to two main findings. First, it enables an efficient applied nodal diagonal connection (ANDC) pruning technique without affecting the accuracy. Second, based on the SNP, spontaneous symmetry breaking occurs among the MHA heads, such that each head focuses its attention on a subset of labels through cooperation among its SNPs. Consequently, each head becomes an expert in recognizing its designated labels, representing a quantitative MHA modus vivendi mechanism. These results are based on a compact convolutional transformer architecture trained on the CIFAR-100 and Flowers-102 datasets and call for their extension to other architectures and applications, such as natural language processing.

Problem

• Convolutional Neural Networks
• Vision Transformers
• Multi-head Attention

Innovation

• Unified Learning Principle
• Efficient Pruning Method
• Node Specialization