Speeding-Up Back-Propagation in DNN: Approximate Outer Product with Memory
In this paper, an algorithm for approximate evaluation of back-propagation in DNN training is considered, which we term Approximate Outer Product Gradient Descent with Memory (Mem-AOP-GD). The Mem-AOP-GD algorithm implements an approximation of the stochastic gradient descent by considering only a subset of the outer products involved in the matrix multiplications that encompass backpropagation. In order to correct for the inherent bias in this approximation, the algorithm retains in memory an accumulation of the outer products that are not used in the approximation. We investigate the performance of the proposed algorithm in terms of DNN training loss under two design parameters: (i) the number of outer products used for the approximation, and (ii) the policy used to select such outer products. We experimentally show that significant improvements in computational complexity as well as accuracy can indeed be obtained through Mem-AOPGD.
Code (1)
Similar Papers 제목 키워드 기반
Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation
In this work we introduce methods to reduce the computational and memory costs of training deep neural networks. Our approach consists in replacing exact vector-jacobian products by randomized, unbiased approximations th…
OrcaRouter: A Production-Oriented LLM Router with Hybrid Offline-Online Learning
The rapid development of large language models, each with distinct capabilities and inference costs, raises a practical deployment question: given an incoming request, which model should handle it? We present OrcaRouter,…
An Approximate Backpropagation Learning Rule for Memristor Based Neural Networks Using Synaptic Plasticity
We describe an approximation to backpropagation algorithm for training deep neural networks, which is designed to work with synapses implemented with memristors. The key idea is to represent the values of both the input …
Outer Product-based Neural Collaborative Filtering
In this work, we contribute a new multi-layer neural network architecture named ONCF to perform collaborative filtering. The idea is to use an outer product to explicitly model the pairwise correlations between the dimen…
Collaborative FilteringMulti-Domain Recommender SystemsRecommendation SystemsBilevel Programs Meet Deep Learning: A Unifying View on Inference Learning Methods
In this work we unify a number of inference learning methods, that are proposed in the literature as alternative training algorithms to the ones based on regular error back-propagation. These inference learning methods w…
Bilevel Optimization