Random active path model of deep neural networks with diluted binary synapses
Deep learning has become a powerful and popular tool for a variety of machine learning tasks. However, it is challenging to understand the mechanism of deep learning from a theoretical perspective. In this work, we propose a random active path model to study collective properties of deep neural networks with binary synapses, under the removal perturbation of connections between layers. In the model, the path from input to output is randomly activated, and the corresponding input unit constrains the weights along the path into the form of a $p$-weight interaction glass model. A critical value of the perturbation is observed to separate a spin glass regime from a paramagnetic regime, with the transition being of the first order. The paramagnetic phase is conjectured to have a poor generalization performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningSimilar Papers 제목 키워드 기반
Learning Spike time codes through Morphological Learning with Binary Synapses
In this paper, a neuron with nonlinear dendrites (NNLD) and binary synapses that is able to learn temporal features of spike input patterns is considered. Since binary synapses are considered, learning happens through fo…
Training Restricted Boltzmann Machines with Binary Synapses using the Bayesian Learning Rule
Restricted Boltzmann machines (RBMs) with low-precision synapses are much appealing with high energy efficiency. However, training RBMs with binary synapses is challenging due to the discrete nature of synapses. Recently…
Variational InferenceSubdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
We show that discrete synaptic weights can be efficiently used for learning in large scale neural systems, and lead to unanticipated computational performance. We focus on the representative case of learning random patte…
Origin of the computational hardness for learning with binary synapses
Supervised learning in a binary perceptron is able to classify an extensive number of random patterns by a proper assignment of binary synaptic weights. However, to find such assignments in practice, is quite a nontrivia…
How do neurons operate on sparse distributed representations? A mathematical theory of sparsity, neurons and active dendrites
We propose a formal mathematical model for sparse representations and active dendrites in neocortex. Our model is inspired by recent experimental findings on active dendritic processing and NMDA spikes in pyramidal neuro…