Dynamic neuron approach to deep neural networks: Decoupling neurons for renormalization group analysis
Deep neural network architectures often consist of repetitive structural elements. We introduce an approach that reveals these patterns and can be broadly applied to the study of deep learning. Similarly to how a power strip helps untangle and organize complex cable connections, this approach treats neurons as additional degrees of freedom in interactions, simplifying the structure and enhancing the intuitive understanding of interactions within deep neural networks. Furthermore, it reveals the translational symmetry of deep neural networks, which simplifies the application of the renormalization group transformation-a method that effectively analyzes the scaling behavior of the system. By utilizing translational symmetry and renormalization group transformations, we can analyze critical phenomena. This approach may open new avenues for studying deep neural networks using statistical physics.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Lecture Notes on Statistical Physics and Neural Networks
These lecture notes introduce some topics of classical statistical physics, particularly those that are relevant for neural networks and deep learning. Statistical physics is treated as a branch of probability theory or …
Phase transitions in in vivo or in vitro populations of spiking neurons belong to different universality classes
The "critical brain hypothesis" posits that neural circuitry may be tuned close to a "critical point" or "phase transition" -- a boundary between different operating regimes of the circuit. The renormalization group and …
Probabilistic models, compressible interactions, and neural coding
In physics we often use very simple models to describe systems with many degrees of freedom, but it is not clear why or how this success can be transferred to the more complex biological context. We consider models for t…
Is Deep Learning a Renormalization Group Flow?
Although there has been a rapid development of practical applications, theoretical explanations of deep learning are in their infancy. Deep learning performs a sophisticated coarse graining. Since coarse graining is a ke…
Deep LearningThe Quenching-Activation Behavior of the Gradient Descent Dynamics for Two-layer Neural Network Models
A numerical and phenomenological study of the gradient descent (GD) algorithm for training two-layer neural network models is carried out for different parameter regimes when the target function can be accurately approxi…