One-step corrected projected stochastic gradient descent for statistical estimation
A generic, fast and asymptotically efficient method for parametric estimation is described. It is based on the projected stochastic gradient descent on the log-likelihood function corrected by a single step of the Fisher scoring algorithm. We show theoretically and by simulations that it is an interesting alternative to the usual stochastic gradient descent with averaging or the adaptative stochastic gradient descent.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Scaling transition from momentum stochastic gradient descent to plain stochastic gradient descent
The plain stochastic gradient descent and momentum stochastic gradient descent have extremely wide applications in deep learning due to their simple settings and low computational complexity. The momentum stochastic grad…
Exponential Concentration in Stochastic Approximation
We analyze the behavior of stochastic approximation algorithms where iterates, in expectation, progress towards an objective at each step. When progress is proportional to the step size of the algorithm, we prove exponen…
SGD at the Edge of Stability: The Stochastic Sharpness Gap
When training neural networks with full-batch gradient descent (GD) and step size $η$, the largest eigenvalue of the Hessian -- the sharpness $S(\boldsymbolθ)$ -- rises to $2/η$ and hovers there, a phenomenon termed the …
Parameter-free projected gradient descent
We consider the problem of minimizing a convex function over a closed convex set, with Projected Gradient Descent (PGD). We propose a fully parameter-free version of AdaGrad, which is adaptive to the distance between the…
Stochastic OptimizationProvable Constrained Stochastic Convex Optimization with XOR-Projected Gradient Descent
Provably solving stochastic convex optimization problems with constraints is essential for various problems in science, business, and statistics. Recently proposed XOR-Stochastic Gradient Descent (XOR-SGD) provides a con…
Management