Computing Linear Regions in Neural Networks with Skip Connections

2025-09-18

AI Summary

This work investigates the mechanistic impact of skip connections on the number of linear regions in deep neural networks. For piecewise-linear activation networks, we propose an exact modeling framework grounded in tropical algebra, wherein forward propagation is represented as compositions of tropical polynomials, and design an efficient algorithm to analytically compute the total number of linear regions across the entire network. Theoretical analysis and empirical evaluation demonstrate that skip connections substantially increase the number of linear regions, and this growth exhibits structural interpretability: it stems from mitigating gradient degradation and enhancing functional expressivity, thereby improving training stability and suppressing overfitting. To our knowledge, this is the first work to establish a quantitative link between skip connections and the geometric complexity of linear regions—measured via their count—providing novel evidence from tropical geometry for the generalization advantage of residual architectures.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Computer Vision: Other Foundations of Computer Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Neural networks are important tools in machine learning. Representing piecewise linear activation functions with tropical arithmetic enables the application of tropical geometry. Algorithms are presented to compute regions where the neural networks are linear maps. Through computational experiments, we provide insights on the difficulty to train neural networks, in particular on the problems of overfitting and on the benefits of skip connections.

Problem

• Computing linear regions in neural networks with skip connections
• Applying tropical geometry to analyze piecewise linear activation functions
• Investigating overfitting problems and benefits of skip connections

Innovation

• Using tropical geometry for activation functions
• Algorithms compute neural network linear regions
• Skip connections improve training and reduce overfitting