Function changes in the Backpropagation equation is equivalent to an implicit learning rate
The backpropagation algorithm is the de-facto standard for credit assignment in artificial neural networks due to its empirical results. Since its conception, variants of the backpropagation algorithm have emerged. More specifically, variants that leverage function changes in the backpropagation equations to satisfy their specific requirements. Feedback Alignment is one such example, which replaces the weight transpose matrix in the backpropagation equations with a random matrix in search of a more biologically plausible credit assignment algorithm. In this work, we show that function changes in the backpropagation procedure is equivalent to adding an implicit learning rate to an artificial neural network. Furthermore, we learn activation function derivatives in the backpropagation equations to demonstrate early convergence in these artificial neural networks. Our work reports competitive performances with early convergence on MNIST and CIFAR10 on sufficiently large deep neural network architectures.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
JFB: Jacobian-Free Backpropagation for Implicit Networks
A promising trend in deep learning replaces traditional feedforward networks with implicit networks. Unlike traditional networks, implicit networks solve a fixed point equation to compute inferences. Solving for the fixe…
Efficient Training of Deep Equilibrium Models
Deep equilibrium models (DEQs) have proven to be very powerful for learning data representations. The idea is to replace traditional (explicit) feedforward neural networks with an implicit fixed-point equation, which all…
Avoiding overfitting of multilayer perceptrons by training derivatives
Resistance to overfitting is observed for neural networks trained with extended backpropagation algorithm. In addition to target values, its cost function uses derivatives of those up to the $4^{\mathrm{th}}$ order. For …
Implicit function theorem for nonlinear time-delay systems with algebraic constraints
In this note, we discuss a generalization of the well-known implicit function theorem to the time-delay case. We show that the latter problem is closely related to the bicausal changes of coordinates of time-delay system…
Learning Neural Event Functions for Ordinary Differential Equations
The existing Neural ODE formulation relies on an explicit knowledge of the termination time. We extend Neural ODEs to implicitly defined termination criteria modeled by neural event functions, which can be chained togeth…
Point Processes