ELiSH

2000년 도입 · 논문 1편에서 사용

The Exponential Linear Squashing Activation Function, or ELiSH, is an activation function used for neural networks. It shares common properties with Swish, being made up of an ELU and a Sigmoid: $$f\left(x\right) = \frac{x}{1+e^{-x}} \text{ if } x \geq 0 $$ $$f\left(x\right) = \frac{e^{x} - 1}{1+e^{-x}} \text{ if } x < 0 $$ The Sigmoid part of ELiSH improves information flow, while the linear parts solve issues of vanishing gradients.

출처: The Quest for the Golden Activation Function

소개 논문: The Quest for the Golden Activation Function