Dual Rectified Linear Units (DReLUs): A Replacement for Tanh Activation Functions in Quasi-Recurrent Neural Networks

2017-07-25 · Fréderic Godin, Jonas Degrave, Joni Dambre, Wesley De Neve

In this paper, we introduce a novel type of Rectified Linear Unit (ReLU), called a Dual Rectified Linear Unit (DReLU). A DReLU, which comes with an unbounded positive and negative image, can be used as a drop-in replacement for a tanh activation function in the recurrent step of Quasi-Recurrent Neural Networks (QRNNs) (Bradbury et al. (2017)). Similar to ReLUs, DReLUs are less prone to the vanishing gradient problem, they are noise robust, and they induce sparse activations. We independently reproduce the QRNN experiments of Bradbury et al. (2017) and compare our DReLU-based QRNNs with the original tanh-based QRNNs and Long Short-Term Memory networks (LSTMs) on sentiment classification and word-level language modeling. Additionally, we evaluate on character-level language modeling, showing that we are able to stack up to eight QRNN layers with DReLUs, thus making it possible to improve the current state-of-the-art in character-level language modeling over shallow architectures based on LSTMs.

📄 PDF Abstract BibTeX arXiv:1707.08214

Code (2)

FredericGodin/QuasiRNN-DReLU 공식 구현

Edresson/VoiceSplit pytorch

Tasks

Language ModelingLanguage Modelling Sentiment AnalysisSentiment Classification

Methods 이 논문이 사용한 방법론

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Sigmoid Activation 설명 없음

Masked Convolution A Masked Convolution is a type of convolution which masks certain pixels so that the model can only predict based on pixels…

QRNN 설명 없음

Tanh Activation 설명 없음

Improving Deep Neural Network with Multiple Parametric Exponential Linear Units

2016-06-01 · Yang Li, Chunxiao Fan, Yong Li, Qiong Wu 외

Activation function is crucial to the recent successes of deep neural networks. In this paper, we first propose a new activation function, Multiple Parametric Exponential Linear Units (MPELU), aiming to generalize and un…

Parametric Rectified Power Sigmoid Units: Learning Nonlinear Neural Transfer Analytical Forms

2021-01-25 · Abdourrahmane Mahamane Atto, Sylvie Galichet, Dominique Pastor, Nicolas Méger

The paper proposes representation functionals in a dual paradigm where learning jointly concerns both linear convolutional weights and parametric forms of nonlinear activation functions. The nonlinear forms proposed for …

BIG-bench Machine Learning

Empirical Evaluation of Rectified Activations in Convolutional Network

2015-05-05 · Bing Xu, Naiyan Wang, Tianqi Chen, Mu Li

In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametr…

General Classification image-classification Image Classification

ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets

2018-05-22 · Narendra Patwardhan, Madhura Ingalhalikar, Rahee Walambe

This work introduces a novel activation unit that can be efficiently employed in deep neural nets (DNNs) and performs significantly better than the traditional Rectified Linear Units (ReLU). The function developed is a t…

FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks

2017-06-25 · Suo Qiu, Xiangmin Xu, Bolun Cai

Rectified linear unit (ReLU) is a widely used activation function for deep convolutional neural networks. However, because of the zero-hard rectification, ReLU networks miss the benefits from negative values. In this pap…

image-classification Image Classification