Accelerating Optimization using Neural Reparametrization
We tackle the problem of accelerating certain optimization problems related to steady states in ODE and energy minimization problems common in physics. We reparametrize the optimization variables as the output of a neural network. We then find the conditions under which this neural reparameterization could speed up convergence rates during gradient descent. We find that to get the maximum speed up the neural network needs to be a special graph convolutional network (GCN) with its aggregation function constructed from the gradients of the loss function. We show the utility of our method on two different optimization problems on graphs and point-clouds.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
The Geometry of Neural Nets' Parameter Spaces Under Reparametrization
Model reparametrization, which follows the change-of-variable rule of calculus, is a popular way to improve the training of neural nets. But it can also be problematic since it can induce inconsistencies in, e.g., Hessia…
Super-Reparametrizations of Weighted CSPs: Properties and Optimization Perspective
The notion of reparametrizations of Weighted CSPs (WCSPs) (also known as equivalence-preserving transformations of WCSPs) is well-known and finds its use in many algorithms to approximate or bound the optimal WCSP value.…
ARCNonlinear Kalman Filtering with Reparametrization Gradients
We introduce a novel nonlinear Kalman filter that utilizes reparametrization gradients. The widely used parametric approximation is based on a jointly Gaussian assumption of the state-space model, which is in turn equiva…
Faster Optimization on Sparse Graphs via Neural Reparametrization
In mathematical optimization, second-order Newton's methods generally converge faster than first-order methods, but they require the inverse of the Hessian, hence are computationally expensive. However, we discover that …
Algorithm to find new identifiable reparametrizations of parametric rational ODE models
Structural identifiability concerns the question of which unknown parameters of a model can be recovered from (perfect) input-output data. If all of the parameters of a model can be recovered from data, the model is said…