Policy Gradient For Multidimensional Action Spaces: Action Sampling and Entropy Bonus

2018-01-01 · ICLR 2018 1 · Vuong Ho Quan, Yiming Zhang, Kenny Song, Xiao-Yue Gong, Keith W. Ross

In recent years deep reinforcement learning has been shown to be adept at solving sequential decision processes with high-dimensional state spaces such as in the Atari games. Many reinforcement learning problems, however, involve high-dimensional discrete action spaces as well as high-dimensional state spaces. In this paper, we develop a novel policy gradient methodology for the case of large multidimensional discrete action spaces. We propose two approaches for creating parameterized policies: LSTM parameterization and a Modified MDP (MMDP) giving rise to Feed-Forward Network (FFN) parameterization. Both of these approaches provide expressive models to which backpropagation can be applied for training. We then consider entropy bonus, which is typically added to the reward function to enhance exploration. In the case of high-dimensional action spaces, calculating the entropy and the gradient of the entropy requires enumerating all the actions in the action space and running forward and backpropagation for each action, which may be computationally infeasible. We develop several novel unbiased estimators for the entropy bonus and its gradient. Finally, we test our algorithms on two environments: a multi-hunter multi-rabbit grid game and a multi-agent multi-arm bandit problem.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Atari GamesDeep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

Efficient Entropy for Policy Gradient with Multidimensional Action Space

2018-06-02 · Yiming Zhang, Quan Ho Vuong, Kenny Song, Xiao-Yue Gong 외

In recent years, deep reinforcement learning has been shown to be adept at solving sequential decision processes with high-dimensional state spaces such as in the Atari games. Many reinforcement learning problems, howeve…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Discrete Action On-Policy Learning with Action-Value Critic

2020-02-10 · Yuguang Yue, Yunhao Tang, Mingzhang Yin, Mingyuan Zhou

Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension, making it challenging to apply existing on-policy gra…

OpenAI Gym Reinforcement Learning Reinforcement Learning (RL)

Learning in complex action spaces without policy gradients

2024-10-08 · Arash Tavakoli, Sina Ghiassian, Nemanja Rakićević

Conventional wisdom suggests that policy gradient methods are better suited to complex action spaces than action-value methods. However, foundational studies have shown equivalences between these paradigms in small and f…

Policy Gradient Methods Q-Learning

Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learning

2023-11-26 · NeurIPS 2023 11 · Changyu Chen, Ramesha Karunasena, Thanh Hong Nguyen, Arunesh Sinha 외

Many problems in Reinforcement Learning (RL) seek an optimal policy with large discrete multidimensional yet unordered action spaces; these include problems in randomized allocation of resources such as placements of mul…

reinforcement-learning Reinforcement Learning (RL)valid

Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking

2024-06-06 · Roland Stolz, Hanna Krasowski, Jakob Thumm, Michael Eichelbeck 외

Continuous action spaces in reinforcement learning (RL) are commonly defined as multidimensional intervals. While intervals usually reflect the action boundaries for tasks well, they can be challenging for learning becau…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)