LayerDrop

2000년 도입 · 논문 4편에서 사용

LayerDrop is a form of structured dropout for Transformer models which has a regularization effect during training and allows for efficient pruning at inference time. It randomly drops layers from the Transformer according to an "every other" strategy where pruning with a rate $p$ means dropping the layers at depth $d$ such that $d = 0\left\(\text{mod}\left(\text{floor}\left(\frac{1}{p}\right)\right)\right)$.

출처: Reducing Transformer Depth on Demand with Structured Dropout

소개 논문: Reducing Transformer Depth on Demand with Structured Dropout