ReGLU

2000년 도입 · 논문 2편에서 사용

ReGLU is an activation function which is a variant of GLU. The definition is as follows: $$ \text{ReGLU}\left(x, W, V, b, c\right) = \max\left(0, xW + b\right) \otimes \left(xV + c\right) $$

출처: GLU Variants Improve Transformer

소개 논문: GLU Variants Improve Transformer