Revisiting Deep Information Propagation: Fractal Frontier and Finite-size Effects
Information propagation characterizes how input correlations evolve across layers in deep neural networks. This framework has been well studied using mean-field theory, which assumes infinitely wide networks. However, these assumptions break down for practical, finite-size networks. In this work, we study information propagation in randomly initialized neural networks with finite width and reveal that the boundary between ordered and chaotic regimes exhibits a fractal structure. This shows the fundamental complexity of neural network dynamics, in a setting that is independent of input data and optimization. To extend this analysis beyond multilayer perceptrons, we leverage recently introduced Fourier-based structured transforms, and show that information propagation in convolutional neural networks also follow the same behavior. In practice, our investigation highlights the importance of finite network depth with respect to the tradeoff between separation and robustness.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
SoFAr: Shortcut-based Fractal Architectures for Binary Convolutional Neural Networks
Binary Convolutional Neural Networks (BCNNs) can significantly improve the efficiency of Deep Convolutional Neural Networks (DCNNs) for their deployment on resource-constrained platforms, such as mobile and embedded syst…
BinarizationThe Fractal Neural Operator: Overcoming Spectral Bias in Chaotic Attractors via Prime-Harmonic Weierstrass Encodings
Deep learning models, particularly Transformers and Neural Operators, exhibit a well-documented "spectral bias," effectively acting as low-pass filters that smooth out high-frequency information. While benign in fluid dy…
FractalBench: Diagnosing Visual-Mathematical Reasoning Through Recursive Program Synthesis
Mathematical reasoning requires abstracting symbolic rules from visual patterns -- inferring the infinite from the finite. We investigate whether multimodal AI systems possess this capability through FractalBench, a benc…
Mathematical ReasoningProgram SynthesisAre Graph Transformers Necessary? Efficient Long-Range Message Passing with Fractal Nodes in MPNNs
Graph Neural Networks (GNNs) have emerged as powerful tools for learning on graph-structured data, but often struggle to balance local and global information. While graph Transformers aim to address this by enabling long…
Computational Efficiencygraph partitioningHow to Sample High Quality 3D Fractals for Action Recognition Pre-Training?
Synthetic datasets are being recognized in the deep learning realm as a valuable alternative to exhaustively labeled real data. One such synthetic data generation method is Formula Driven Supervised Learning (FDSL), whic…
Synthetic Data GenerationAction Recognition