How to train your neural ODE: the world of Jacobian and kinetic regularization
Training neural ODEs on large datasets has not been tractable due to the necessity of allowing the adaptive numerical ODE solver to refine its step size to very small values. In practice this leads to dynamics equivalent to many hundreds or even thousands of layers. In this paper, we overcome this apparent difficulty by introducing a theoretically-grounded combination of both optimal transport and stability regularizations which encourage neural ODEs to prefer simpler dynamics out of all the dynamics that solve a problem well. Simpler dynamics lead to faster convergence and to fewer discretizations of the solver, considerably decreasing wall-clock time without loss in performance. Our approach allows us to train neural ODE-based generative models to the same performance as the unregularized dynamics, with significant reductions in training time. This brings neural ODEs closer to practical relevance in large-scale applications.
Code (2)
Tasks
Density EstimationImage GenerationSimilar Papers 제목 키워드 기반
Adversarially robust generalization theory via Jacobian regularization for deep neural networks
Powerful deep neural networks are vulnerable to adversarial attacks. To obtain adversarially robust models, researchers have separately developed adversarial training and Jacobian regularization techniques. There are abu…
Adversarial AttackBridging Optimal Transport and Jacobian Regularization by Optimal Trajectory for Enhanced Adversarial Defense
Deep neural networks, particularly in vision tasks, are notably susceptible to adversarial perturbations. To overcome this challenge, developing a robust classifier is crucial. In light of the recent advancements in the …
Adversarial AttackAdversarial DefenseAdversarial RobustnessIsokinetic Flow Matching for Pathwise Straightening of Generative Flows
Flow Matching (FM) constructs linear conditional probability paths, but the learned marginal velocity field inevitably exhibits strong curvature due to trajectory superposition. This curvature severely inflates numerical…
Generalizing and Improving Jacobian and Hessian Regularization
Jacobian and Hessian regularization aim to reduce the magnitude of the first and second-order partial derivatives with respect to neural network inputs, and they are predominantly used to ensure the adversarial robustnes…
Adversarial RobustnessConditioning of Reinforcement Learning Agents and its Policy Regularization Application
The outcome of Jacobian singular values regularization was studied for supervised learning problems. It also was shown that Jacobian conditioning regularization can help to avoid the ``mode-collapse'' problem in Generati…
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning+1