Efficient Numerical Algorithm for Large-Scale Damped Natural Gradient Descent
We propose a new algorithm for efficiently solving the damped Fisher matrix in large-scale scenarios where the number of parameters significantly exceeds the number of available samples. This problem is fundamental for natural gradient descent and stochastic reconfiguration. Our algorithm is based on Cholesky decomposition and is generally applicable. Benchmark results show that the algorithm is significantly faster than existing methods.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Accelerating Langevin Monte Carlo Sampling: A Large Deviations Analysis
Langevin algorithms are popular Markov chain Monte Carlo methods that are often used to solve high-dimensional large-scale sampling problems in machine learning. The most classical Langevin Monte Carlo algorithm is based…
Adaptive Hybrid Subspace Levenberg Marquardt Algorithm with Adequacy Monitor for Large Scale Least Squares Problems
The Levenberg-Marquardt (LM) algorithm is the most widely used method for solving nonlinear least-squares problems, as it combines the robustness of steepest descent with the fast local convergence of the Gauss-Newton me…
Ergodicity of the underdamped mean-field Langevin dynamics
We study the long time behavior of an underdamped mean-field Langevin (MFL) equation, and provide a general convergence as well as an exponential convergence rate result under different conditions. The results on the MFL…
From characteristic functions to multivariate distribution functions and European option prices by the damped COS method
We provide a unified framework to obtain numerically certain quantities, such as the distribution function, absolute moments and prices of financial options, from the characteristic function of some (unknown) probability…
Incorporating Damped Harmonic Oscillator in DSGE Models
This paper integrates the damped harmonic oscillator into DSGE models to better capture delayed economic adjustments. By introducing a damping coefficient, I model economic recoveries as under-damped, critically damped, …