Neural Network Regularization via Robust Weight Factorization
Regularization is essential when training large neural networks. As deep neural networks can be mathematically interpreted as universal function approximators, they are effective at memorizing sampling noise in the training data. This results in poor generalization to unseen data. Therefore, it is no surprise that a new regularization technique, Dropout, was partially responsible for the now-ubiquitous winning entry to ImageNet 2012 by the University of Toronto. Currently, Dropout (and related methods such as DropConnect) are the most effective means of regularizing large neural networks. These amount to efficiently visiting a large number of related models at training time, while aggregating them to a single predictor at test time. The proposed FaMe model aims to apply a similar strategy, yet learns a factorization of each weight matrix such that the factors are robust to noise.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries
Sparse regularization techniques are well-established in machine learning, yet their application in neural networks remains challenging due to the non-differentiability of penalties like the $L_1$ norm, which is incompat…
Sparse LearningDoes Weighting Improve Matrix Factorization for Recommender Systems?
Matrix factorization is a widely used approach for top-N recommendation and collaborative filtering. When implemented on implicit feedback data (such as clicks), a common heuristic is to upweight the observed interaction…
Collaborative FilteringB3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learning
Overestimation arising from selecting unseen actions during policy evaluation is a major challenge in offline reinforcement learning (RL). A minimalist approach in the single-agent setting -- adding behavior cloning (BC)…
Multi-agent Reinforcement LearningReinforcement Learning (RL)An Iterative Algorithm for Regularized Non-negative Matrix Factorizations
We generalize the non-negative matrix factorization algorithm of Lee and Seung to accept a weighted norm, and to support ridge and Lasso regularization. We recast the Lee and Seung multiplicative update as an additive up…
Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization
Quantization of neural networks has become common practice, driven by the need for efficient implementations of deep neural networks on embedded devices. In this paper, we exploit an oft-overlooked degree of freedom in m…
Network PruningQuantization