The Outer Product Structure of Neural Network Derivatives
In this paper, we show that feedforward and recurrent neural networks exhibit an outer product derivative structure but that convolutional neural networks do not. This structure makes it possible to use higher-order information without needing approximations or infeasibly large amounts of memory, and it may also provide insights into the geometry of neural network optima. The ability to easily access these derivatives also suggests a new, geometric approach to regularization. We then discuss how this structure could be used to improve training methods, increase network robustness and generalizability, and inform network compression methods.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
POP Image Fusion - Derivative Domain Image Fusion Without Reintegration
There are many applications where multiple images are fused to form a single summary greyscale or colour output, including computational photography (e.g. RGB-NIR), diffusion tensor imaging (medical), and remote sensing.…
The Zero-Coupon Rate Model for Derivatives Pricing
The aim of this paper is to present a dual-term structure model of interest rate derivatives in order to solve the two hardest problems in financial modeling: the exact volatility calibration of the entire swaption matri…
modelEfficiently Access Diffusion Fisher: Within the Outer Product Span Space
Recent Diffusion models (DMs) advancements have explored incorporating the second-order diffusion Fisher information (DF), defined as the negative Hessian of log density, into various downstream tasks and theoretical ana…
Speeding-Up Back-Propagation in DNN: Approximate Outer Product with Memory
In this paper, an algorithm for approximate evaluation of back-propagation in DNN training is considered, which we term Approximate Outer Product Gradient Descent with Memory (Mem-AOP-GD). The Mem-AOP-GD algorithm implem…
Outer Product-based Neural Collaborative Filtering
In this work, we contribute a new multi-layer neural network architecture named ONCF to perform collaborative filtering. The idea is to use an outer product to explicitly model the pairwise correlations between the dimen…
Collaborative FilteringMulti-Domain Recommender SystemsRecommendation Systems