SwiGLU

2000년 도입 · 논문 13편에서 사용

SwiGLU is an activation function which is a variant of GLU. The definition is as follows: $$ \text{SwiGLU}\left(x, W, V, b, c, \beta\right) = \text{Swish}\_{\beta}\left(xW + b\right) \otimes \left(xV + c\right) $$

출처: GLU Variants Improve Transformer

소개 논문: GLU Variants Improve Transformer