Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexity

2026-06-19

AI Summary

This work investigates why trained neural networks utilize far fewer affine regions than their theoretical capacity, particularly exhibiting “expressivity saturation” as task complexity increases. Focusing on piecewise-affine networks such as ReLU MLPs, the study integrates one-dimensional line probe theory, exact counting of high-dimensional affine regions, and visualization of training dynamics to derive an upper bound on active affine segments governed by layer width and activation breakpoints. The analysis reveals that increasing task complexity paradoxically leads to a significant reduction in the number of actually activated regions. Moreover, in high-difficulty tasks, this region collapse frequently coincides with degenerate decision boundaries, uncovering a non-monotonic relationship between model expressivity and task difficulty.

Abstract

Piecewise-affine neural networks (e.g., with ReLU or LeakyReLU activations) implement continuous piecewise-affine maps, and the number of affine regions provides a natural proxy for expressive capacity. However, the gap between theoretical region capacity and the affine regions realized after training remains insufficiently understood. We study this gap from two complementary perspectives. First, we give a rigorous, architecture-dependent theorem for affine line-segment probes: for multilayer perceptrons with piecewise-affine activations, the number of affine pieces realized along an affine line-segment probe is upper bounded by an explicit product of layer-wise width terms (and activation breakpoint factors). This yields a neuron-threshold lower bound for representing target functions with prescribed one-dimensional piece complexity, formalizing the minimal region budget required for complex signals. Second, we exactly enumerate affine regions realized within bounded 2D and higher-dimensional domains under controlled task complexity. Under fixed architectures and training protocols, increasing input--label complexity yields trained solutions with markedly fewer realized regions in the evaluation domain, even though worst-case architectural capacity is unchanged; we call this reduced region usage expressivity saturation. Moreover, in the most challenging regimes, 2D visualizations show that region-usage collapse often coincides with degraded decision boundaries. Finally, we visualize the training dynamics of affine-region partitions and decision boundaries, revealing a consistent refinement process during optimization.

Problem

• expressivity saturation
• affine regions
• piecewise-affine neural networks
• task complexity
• expressive capacity

Innovation

• expressivity saturation
• affine regions
• piecewise-affine neural networks
• region enumeration
• expressive capacity