Activation Regularization

2000년 도입 · 논문 56편에서 사용

Activation Regularization (AR), or $L\_{2}$ activation regularization, is regularization performed on activations as opposed to weights. It is usually used in conjunction with RNNs. It is defined as: $$\alpha{L}\_{2}\left(m\circ{h\_{t}}\right) $$ where $m$ is a dropout mask used by later parts of the model, $L\_{2}$ is the $L\_{2}$ norm, and $h_{t}$ is the output of an RNN at timestep $t$, and $\alpha$ is a scaling coefficient. When applied to the output of a dense layer, AR penalizes activations that are substantially away from 0, encouraging activations to remain small.

출처: Revisiting Activation Regularization for Language RNNs

소개 논문: Revisiting Activation Regularization for Language RNNs