STL: A Signed and Truncated Logarithm Activation Function for Neural Networks
Activation functions play an essential role in neural networks. They provide the non-linearity for the networks. Therefore, their properties are important for neural networks' accuracy and running performance. In this paper, we present a novel signed and truncated logarithm function as activation function. The proposed activation function has significantly better mathematical properties, such as being odd function, monotone, differentiable, having unbounded value range, and a continuous nonzero gradient. These properties make it an excellent choice as an activation function. We compare it with other well-known activation functions in several well-known neural networks. The results confirm that it is the state-of-the-art. The suggested activation function can be applied in a large range of neural networks where activation functions are necessary.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
TSSR: A Truncated and Signed Square Root Activation Function for Neural Networks
Activation functions are essential components of neural networks. In this paper, we introduce a new activation function called the Truncated and Signed Square Root (TSSR) function. This function is distinctive because it…
speech-recognitionSpeech RecognitionNatural-Logarithm-Rectified Activation Function in Convolutional Neural Networks
Activation functions play a key role in providing remarkable performance in deep neural networks, and the rectified linear unit (ReLU) is one of the most widely used activation functions. Various new activation functions…
Shallow neural network approximation in mixed Sobolev spaces
We investigate the best $L_2$ approximation of mixed Sobolev spaces by shallow neural networks with $n$ neurons and general activation functions. We first establish an activation-independent Fourier-block principle: if a…
Most Activation Functions Can Win the Lottery Without Excessive Depth
The strong lottery ticket hypothesis has highlighted the potential for training deep neural networks by pruning, which has inspired interesting practical and theoretical insights into how neural networks can represent fu…
A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks
We present the soft exponential activation function for artificial neural networks that continuously interpolates between logarithmic, linear, and exponential functions. This activation function is simple, differentiable…