Stochastic Weight Averaging

2000년 도입 · 논문 45편에서 사용

Stochastic Weight Averaging is an optimization procedure that averages multiple points along the trajectory of SGD, with a cyclical or constant learning rate. On the one hand it averages weights, but it also has the property that, with a cyclical or constant learning rate, SGD proposals are approximately sampling from the loss surface of the network, leading to stochastic weights and helping to discover broader optima.

출처: Averaging Weights Leads to Wider Optima and Better Generalization

소개 논문: Averaging Weights Leads to Wider Optima and Better Generalization