Explicit Lipschitz Value Estimation Enhances Policy Robustness Against Perturbation
In robotic control tasks, policies trained by reinforcement learning (RL) in simulation often experience a performance drop when deployed on physical hardware, due to modeling error, measurement error, and unpredictable perturbations in the real world. Robust RL methods account for this issue by approximating a worst-case value function during training, but they can be sensitive to approximation errors in the value function and its gradient before training is complete. In this paper, we hypothesize that Lipschitz regularization can help condition the approximated value function gradients, leading to improved robustness after training. We test this hypothesis by combining Lipschitz regularization with an application of Fast Gradient Sign Method to reduce approximation errors when evaluating the value function under adversarial perturbations. Our empirical results demonstrate the benefits of this approach over prior work on a number of continuous control benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Off-Policy Interval Estimation with Lipschitz Value Iteration
Off-policy evaluation provides an essential tool for evaluating the effects of different policies or treatments using only observed data. When applied to high-stakes scenarios such as medical diagnosis or financial decis…
Decision MakingMedical DiagnosisOff-policy evaluationImproving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Action
Existing value-based online reinforcement learning (RL) algorithms suffer from slow policy exploitation due to ineffective exploration and delayed policy updates. To address these challenges, we propose an algorithm call…
Representation LearningReinforcement LearningContinuous ControlCollaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning
Deep off-policy reinforcement learning algorithms for continuous control typically rely on neural value function approximation to guide policy improvement. However, temporal-difference (TD) learning introduces noisy targ…
Reinforcement LearningContinuous ControlImproving Value Estimation Critically Enhances Vanilla Policy Gradient
Modern policy gradient algorithms, such as TRPO and PPO, outperform vanilla policy gradient in many RL tasks. Questioning the common belief that enforcing approximate trust regions leads to steady policy improvement in p…
continuous-controlContinuous ControlNeural Policy Iteration for Stochastic Optimal Control: A Physics-Informed Approach
We propose a physics-informed neural network policy iteration (PINN-PI) framework for solving stochastic optimal control problems governed by second-order Hamilton--Jacobi--Bellman (HJB) equations. At each iteration, a n…