Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)
We propose a novel activation function that implements piece-wise orthogonal non-linear mappings based on permutations. It is straightforward to implement, and very computationally efficient, also it has little memory requirements. We tested it on two toy problems for feedforward and recurrent networks, it shows similar performance to tanh and ReLU. OPLU activation function ensures norm preservance of the backpropagated gradients, therefore it is potentially good for the training of deep, extra deep, and recurrent neural networks.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
AuON: A Linear-time Alternative to Orthogonal Momentum Updates
Orthogonal momentum gradient updates have emerged to overcome the limitations of vector-based optimizers like Adam. The vector-based optimizer Adam suffers from high memory costs and ill-conditioned momentum gradient upd…
Complete Dictionary Learning via $\ell^4$-Norm Maximization over the Orthogonal Group
This paper considers the fundamental problem of learning a complete (orthogonal) dictionary from samples of sparsely generated signals. Most existing methods solve the dictionary (and sparse representations) based on heu…
Dictionary LearningDizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation
The vanishing and exploding gradient problems are well-studied obstacles that make it difficult for recurrent neural networks to learn long-term time dependencies. We propose a reparameterization of standard recurrent ne…
Robust Permutation Tests in Linear Instrumental Variables Regression
This paper develops permutation versions of identification-robust tests in linear instrumental variables (IV) regression. Unlike the existing randomization and rank-based tests in which independence between the instrumen…
regressionQuery Complexity of Active Learning for Function Family With Nearly Orthogonal Basis
Many machine learning algorithms require large numbers of labeled data to deliver state-of-the-art results. In applications such as medical diagnosis and fraud detection, though there is an abundance of unlabeled data, i…
Active LearningFraud DetectionMedical Diagnosisregression