Linear Warmup With Linear Decay

2000년 도입 · 논문 7,076편에서 사용

Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.