Revisiting Implicit Differentiation for Learning Problems in Optimal Control
This paper proposes a new method for differentiating through optimal trajectories arising from non-convex, constrained discrete-time optimal control (COC) problems using the implicit function theorem (IFT). Previous works solve a differential Karush-Kuhn-Tucker (KKT) system for the trajectory derivative, and achieve this efficiently by solving an auxiliary Linear Quadratic Regulator (LQR) problem. In contrast, we directly evaluate the matrix equations which arise from applying variable elimination on the Lagrange multiplier terms in the (differential) KKT system. By appropriately accounting for the structure of the terms within the resulting equations, we show that the trajectory derivatives scale linearly with the number of timesteps. Furthermore, our approach allows for easy parallelization, significantly improved scalability with model size, direct computation of vector-Jacobian products and improved numerical stability compared to prior works. As an additional contribution, we unify prior works, addressing claims that computing trajectory derivatives using IFT scales quadratically with the number of timesteps. We evaluate our method on a both synthetic benchmark and four challenging, learning from demonstration benchmarks including a 6-DoF maneuvering quadrotor and 6-DoF rocket powered landing.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Efficient and Modular Implicit Differentiation
Automatic differentiation (autodiff) has revolutionized machine learning. It allows to express complex computations by composing elementary ones in creative ways and removes the burden of computing their derivatives by h…
Meta-LearningImproved Marginal Unbiased Score Expansion (MUSE) via Implicit Differentiation
We apply the technique of implicit differentiation to boost performance, reduce numerical error, and remove required user-tuning in the Marginal Unbiased Score Expansion (MUSE) algorithm for hierarchical Bayesian inferen…
Bayesian InferenceProbabilistic ProgrammingVectorized Sparse Second-Order Forward Automatic Differentiation for Optimal Control Direct Methods
Direct collocation methods are widely used numerical techniques for solving optimal control problems. The discretization of continuous-time optimal control problems transforms them into large-scale nonlinear programming …
Computational EfficiencyEfficient differentiable quadratic programming layers: an ADMM approach
Recent advances in neural-network architecture allow for seamless integration of convex optimization problems as differentiable layers in an end-to-end trainable neural network. Integrating medium and large scale quadrat…
Portfolio OptimizationControl-Oriented Model-Based Reinforcement Learning with Implicit Differentiation
The shortcomings of maximum likelihood estimation in the context of model-based reinforcement learning have been highlighted by an increasing number of papers. When the model class is misspecified or has a limited repres…
Model-based Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)