DISE: Dynamic Integrator Selection to Minimize Forward Pass Time in Neural ODEs
Neural ordinary differential equations (Neural ODEs) are appreciated for their ability to significantly reduce the number of parameters when constructing a neural network. On the other hand, they are blamed for their procrastinated forward-pass inference, which is incurred by solving integral problems. To improve the model accuracy, they rely on advanced solvers, such as the Dormand--Prince (DOPRI) method. To solve an integral problem, however, it requires at least tens (or sometimes hundreds) of steps in many Neural ODE experiments. In this work, we propose to i) directly regularize the step size of DOPRI to make the forward-pass faster and ii) dynamically choose a simpler integrator than DOPRI for a carefully selected subset of input. Because it is not the case that every input requires the advanced integrator, we design an auxiliary neural network to choose an appropriate integrator given input to decrease the overall inference time without significantly sacrificing accuracy. We consider the Euler method, the fourth-order Runge--Kutta (RK4) method, and DOPRI as selection candidates. We found that a non-trivial percentage of cases can be solved with simple integrators. Therefore, the overall number of functional evaluations (NFEs) decreases up to 78% with improved accuracy.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Integrator Forwading Design for Unicycles with Constant and Actuated Velocity in Polar Coordinates
In a companion paper, we present a modular framework for unicycle stabilization in polar coordinates that provides smooth steering laws through backstepping. Surprisingly, the same problem also allows the application of …
An Exact Bitwise Reversible Integrator
At a fundamental level most physical equations are time reversible. In this paper we propose an integrator that preserves this property at the discrete computational level. Our simulations can be run forward and backward…
Learned Lagrangian Models of PDEs via Euler-Lagrange Residual Minimization
We present the first method to directly use a learned continuous Lagrangian to forecast the dynamics of systems governed by partial differential equations, exploiting the inherent conservative structure to achieve stable…
Multistep Frequency Response Optimized Integrators and Their Application to Accelerating a Power System Transient Simulation Scheme
This paper proposes several explicit and implicit multistep frequency response optimized integrators considering first or second order derivative. A prediction-based method aiming at accelerating a novel power system tra…
PredictionBearing-Only Solution for Fermat-Weber Location Problem: Generalized Algorithms
This paper presents novel algorithms for the Fermat-Weber Location Problem, guiding an autonomous agent to the point that minimizes the weighted sum of Euclidean distances to some beacons using only bearing measurements.…