Revisiting Deep Information Propagation: Fractal Frontier and Finite-size Effects

2025-08-05

AI Summary

This study investigates information propagation dynamics in finite-width randomly initialized neural networks, aiming to characterize the geometric nature of the order-to-chaos transition boundary. Using tools from random matrix theory and Fourier-structured transformations, we establish— for the first time—that this phase boundary exhibits a universal fractal structure, independent of input distribution and training dynamics. This finding transcends the infinite-width assumption inherent in classical mean-field theory, demonstrating that fractal criticality is an intrinsic property of finite-scale networks. We extend the analysis to convolutional neural networks (CNNs), confirming identical propagation laws and thereby unifying information propagation characterization across fully connected and convolutional architectures. Moreover, we elucidate how network depth fundamentally governs the trade-off between representation separability and adversarial robustness. Collectively, our work establishes a new paradigm for understanding the essential complexity of finite-width neural networks.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Non-convex Optimization

Application Category

• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Information propagation characterizes how input correlations evolve across layers in deep neural networks. This framework has been well studied using mean-field theory, which assumes infinitely wide networks. However, these assumptions break down for practical, finite-size networks. In this work, we study information propagation in randomly initialized neural networks with finite width and reveal that the boundary between ordered and chaotic regimes exhibits a fractal structure. This shows the fundamental complexity of neural network dynamics, in a setting that is independent of input data and optimization. To extend this analysis beyond multilayer perceptrons, we leverage recently introduced Fourier-based structured transforms, and show that information propagation in convolutional neural networks also follow the same behavior. Our investigation highlights the importance of finite network depth with respect to the tradeoff between separation and robustness.

Problem

• Study information propagation in finite-width neural networks
• Reveal fractal structure in ordered-chaotic regime boundaries
• Analyze propagation in CNNs using Fourier-based transforms

Innovation

• Studying finite-width neural networks dynamics
• Revealing fractal structure in propagation boundaries
• Extending analysis using Fourier-based transforms