Efficient Entropy for Policy Gradient with Multidimensional Action Space

2018-06-02 · Yiming Zhang, Quan Ho Vuong, Kenny Song, Xiao-Yue Gong, Keith W. Ross

In recent years, deep reinforcement learning has been shown to be adept at solving sequential decision processes with high-dimensional state spaces such as in the Atari games. Many reinforcement learning problems, however, involve high-dimensional discrete action spaces as well as high-dimensional state spaces. This paper considers entropy bonus, which is used to encourage exploration in policy gradient. In the case of high-dimensional action spaces, calculating the entropy and its gradient requires enumerating all the actions in the action space and running forward and backpropagation for each action, which may be computationally infeasible. We develop several novel unbiased estimators for the entropy bonus and its gradient. We apply these estimators to several models for the parameterized policies, including Independent Sampling, CommNet, Autoregressive with Modified MDP, and Autoregressive with LSTM. Finally, we test our algorithms on two environments: a multi-hunter multi-rabbit grid game and a multi-agent multi-arm bandit problem. The results show that our entropy estimators substantially improve performance with marginal additional computational cost.

📄 PDF Abstract BibTeX arXiv:1806.00589

Code (0)

등록된 구현이 없습니다.

Tasks

Atari GamesDeep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

Policy Gradient For Multidimensional Action Spaces: Action Sampling and Entropy Bonus

2018-01-01 · ICLR 2018 1 · Vuong Ho Quan, Yiming Zhang, Kenny Song, Xiao-Yue Gong 외

In recent years deep reinforcement learning has been shown to be adept at solving sequential decision processes with high-dimensional state spaces such as in the Atari games. Many reinforcement learning problems, however…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Entropy annealing for policy mirror descent in continuous time and space

2024-05-30 · Deven Sethi, David Šiška, Yufei Zhang

Entropy regularization has been widely used in policy optimization algorithms to enhance exploration and the robustness of the optimal control; however it also introduces an additional regularization bias. This work quan…

Policy Gradient Methods

Discrete Action On-Policy Learning with Action-Value Critic

2020-02-10 · Yuguang Yue, Yunhao Tang, Mingzhang Yin, Mingyuan Zhou

Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension, making it challenging to apply existing on-policy gra…

OpenAI Gym Reinforcement Learning Reinforcement Learning (RL)

A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces

2023-10-04 · Bekzhan Kerimkulov, James-Michael Leahy, David Siska, Lukasz Szpruch 외

We study the global convergence of a Fisher-Rao policy gradient flow for infinite-horizon entropy-regularised Markov decision processes with Polish state and action space. The flow is a continuous-time analogue of a poli…

LEMMA

Retry Policy Gradients in Continuous Action Spaces

2026-06-04 · Soichiro Nishimori, Paavo Parmas arxiv

Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration without explicit exploration bonuses. In d…