Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization

2023-02-05 · Zichuan Lin, Xiapeng Wu, Mingfei Sun, Deheng Ye, Qiang Fu, Wei Yang, Wei Liu

Recent success in Deep Reinforcement Learning (DRL) methods has shown that policy optimization with respect to an off-policy distribution via importance sampling is effective for sample reuse. In this paper, we show that the use of importance sampling could introduce high variance in the objective estimate. Specifically, we show in a principled way that the variance of importance sampling estimate grows quadratically with importance ratios and the large ratios could consequently jeopardize the effectiveness of surrogate objective optimization. We then propose a technique called sample dropout to bound the estimation variance by dropping out samples when their ratio deviation is too high. We instantiate this sample dropout technique on representative policy optimization algorithms, including TRPO, PPO, and ESPO, and demonstrate that it consistently boosts the performance of those DRL algorithms on both continuous and discrete action controls, including MuJoCo, DMControl and Atari video games. Our code is open-sourced at \url{https://github.com/LinZichuan/sdpo.git}.

📄 PDF Abstract BibTeX arXiv:2302.02299

Code (1)

linzichuan/sdpo 공식 구현 tf

Tasks

Deep Reinforcement LearningMuJoCo

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

TRPO Trust Region Policy Optimization, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL…

Jumpout: Improved Dropout for Deep Neural Networks with Rectified Linear Units

2019-05-01 · ICLR 2019 5 · Shengjie Wang, Tianyi Zhou, Jeff Bilmes

Dropout is a simple yet effective technique to improve generalization performance and prevent overfitting in deep neural networks (DNNs). In this paper, we discuss three novel observations about dropout to better underst…

A PAC-Bayesian Tutorial with A Dropout Bound

2013-07-08 · David McAllester

This tutorial gives a concise overview of existing PAC-Bayesian theory focusing on three generalization bounds. The first is an Occam bound which handles rules with finite precision parameters and which states that gener…

Generalization Bounds

Multi-level Monte Carlo Dropout for Efficient Uncertainty Quantification

2026-01-19 · Aaron Pim, Tristan Pryer arxiv

We develop a multilevel Monte Carlo (MLMC) framework for uncertainty quantification with Monte Carlo dropout. Treating dropout masks as a source of epistemic randomness, we define a fidelity hierarchy by the number of st…

Adjusting for Dropout Variance in Batch Normalization and Weight Initialization

2016-07-08 · Dan Hendrycks, Kevin Gimpel

We show how to adjust for the variance introduced by dropout with corrections to weight initialization and Batch Normalization, yielding higher accuracy. Though dropout can preserve the expected input to a neuron between…

Data Augmentation

Direct Acceleration of SAGA using Sampled Negative Momentum

2018-06-28 · Kaiwen Zhou

Variance reduction is a simple and effective technique that accelerates convex (or non-convex) stochastic optimization. Among existing variance reduction methods, SVRG and SAGA adopt unbiased gradient estimators and are …

Stochastic Optimization