SERLU

2000년 도입 · 논문 1편에서 사용

SERLU, or Scaled Exponentially-Regularized Linear Unit, is a type of activation function. The new function introduces a bump-shaped function in the region of negative input. The bump-shaped function has approximately zero response to large negative input while being able to push the output of SERLU towards zero mean statistically. $$ \text{SERLU}\left(x\right)) = \lambda\_{serlu}x \text{ if } x \geq 0 $$ $$ \text{SERLU}\left(x\right)) = \lambda\_{serlu}\alpha\_{serlu}xe^{x} \text{ if } x < 0 $$ where the two parameters $\lambda\_{serlu} > 0$ and $\alpha\_{serlu} > 0$ remain to be specified.

출처: Effectiveness of Scaled Exponentially-Regularized Linear Units (SERLUs)

소개 논문: Effectiveness of Scaled Exponentially-Regularized Linear Units (SERLUs)