AGLU

2000년 도입 · 논문 1편에서 사용

The Adaptive Parametric activation (APA) is defined as: $APA(z,λ,κ) = (λ exp(−κz) + 1) ^{\frac{1}{−λ}}$, where $λ$ and $κ$ are learnable parameters. This activation function is a generalisation of the Sigmoid and the Gumbel activation functions and it is expressive and versatile. For example, APA can be used inside the channel attention mechanism instead of the Sigmoid activation, or it can be used inside the intermediate layers using the Adaptive Generalised Linear Unit (AGLU): $AGLU(z,λ,κ) = z APA(z,λ,κ)$.

출처: Adaptive Parametric Activation

소개 논문: Adaptive Parametric Activation