Stochastic Depth

2000년 도입 · 논문 463편에서 사용

Stochastic Depth aims to shrink the depth of a network during training, while keeping it unchanged during testing. This is achieved by randomly dropping entire ResBlocks during training and bypassing their transformations through skip connections. Let $b\_{l} \in$ {$0, 1$} denote a Bernoulli random variable, which indicates whether the $l$th ResBlock is active ($b\_{l} = 1$) or inactive ($b\_{l} = 0$). Further, let us denote the “survival” probability of ResBlock $l$ as $p\_{l} = \text{Pr}\left(b\_{l} = 1\right)$. With this definition we can bypass the $l$th ResBlock by multiplying its function $f\_{l}$ with $b\_{l}$ and we extend the update rule to: $$ H\_{l} = \text{ReLU}\left(b\_{l}f\_{l}\left(H\_{l-1}\right) + \text{id}\left(H\_{l-1}\right)\right) $$ If $b\_{l} = 1$, this reduces to the original ResNet update and this ResBlock remains unchanged. If $b\_{l} = 0$, the ResBlock reduces to the identity function, $H\_{l} = \text{id}\left((H\_{l}−1\right)$.

출처: Deep Networks with Stochastic Depth

소개 논문: Deep Networks with Stochastic Depth