Neural networks with dynamical coefficients and adjustable connections on the basis of integrated backpropagation
We consider artificial neurons which will update their weight coefficients with an internal rule based on backpropagation, rather than using it as an external training procedure. To achieve this we include the backpropagation error estimate as a separate entity in all the neuron models and perform its exchange along the synaptic connections. In addition to this we add some special type of neurons with reference inputs, which will serve as a base source of error estimates for the whole network. Finally, we introduce a training control signal for all the neurons, which can enable the correction of weights and the exchange of error estimates. For recurrent neural networks we also demonstrate how to integrate backpropagation through time into their formalism with the help of some stack memory for reference inputs and external data inputs of neurons. Also, for widely used neural networks, such as long short-term memory, radial basis function networks, multilayer perceptrons and convolutional neural networks, we demonstrate their alternative description within the framework of our new formalism.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
ARMA Nets: Expanding Receptive Field for Dense Prediction
Global information is essential for dense prediction problems, whose goal is to compute a discrete or continuous label for each pixel in the images. Traditional convolutional layers in neural networks, initially designed…
image-classificationImage ClassificationPredictionSemantic Segmentation+1Learning Horizontal Connections in a Sparse Coding Model of Natural Images
It has been shown that adapting a dictionary of basis functions to the statistics of natural images so as to maximize sparsity in the coefficients results in a set of dictionary elements whose spatial properties resemble…
A flexible state space model for learning nonlinear dynamical systems
We consider a nonlinear state-space model with the state transition and observation functions expressed as basis function expansions. The coefficients in the basis function expansions are learned from data. Using a conne…
Gaussian ProcessesData-Driven Variational Basis Learning Beyond Neural Networks: A Non-Neural Framework for Adaptive Basis Discovery
Classical representation systems such as Fourier series, wavelets, and fixed dictionaries provide analytically tractable basis expansions, but they are not intrinsically adapted to the empirical structure of modern high-…
Representation LearningBasis-to-Basis Operator Learning Using Function Encoders
We present Basis-to-Basis (B2B) operator learning, a novel approach for learning operators on Hilbert spaces of functions based on the foundational ideas of function encoders. We decompose the task of learning operators …
Operator learning