Linear Warmup

2000년 도입 · 논문 38편에서 사용

Linear Warmup is a learning rate schedule where we linearly increase the learning rate from a low rate to a constant rate thereafter. This reduces volatility in the early stages of training. Image Credit: Chengwei Zhang