On the Linear Convergence of Policy Gradient under Hadamard Parameterization
The convergence of deterministic policy gradient under the Hadamard parameterization is studied in the tabular setting and the linear convergence of the algorithm is established. To this end, we first show that the error decreases at an $O(\frac{1}{k})$ rate for all the iterations. Based on this result, we further show that the algorithm has a faster local linear convergence rate after $k_0$ iterations, where $k_0$ is a constant that only depends on the MDP problem and the initialization. To show the local linear convergence of the algorithm, we have indeed established the contraction of the sub-optimal probability $b_s^k$ (i.e., the probability of the output policy $\pi^k$ on non-optimal actions) when $k\ge k_0$.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Decentralized Online Riemannian Optimization Beyond Hadamard Manifolds
We study decentralized online Riemannian optimization over manifolds with possibly positive curvature, going beyond the Hadamard manifold setting. Decentralized optimization techniques rely on a consensus step that is we…
Elementary Analysis of Policy Gradient Methods
Projected policy gradient under the simplex parameterization, policy gradient and natural policy gradient under the softmax parameterization, are fundamental algorithms in reinforcement learning. There have been a flurry…
Policy Gradient MethodsHLQ: Fast and Efficient Backpropagation via Hadamard Low-rank Quantization
With the rapid increase in model size and the growing importance of various fine-tuning applications, lightweight training has become crucial. Since the backward pass is twice as expensive as the forward pass, optimizing…
QuantizationWhen Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence
Generative adversarial imitation learning (GAIL) is a popular inverse reinforcement learning approach for jointly optimizing policy and reward from expert trajectories. A primary question about GAIL is whether applying a…
Imitation LearningOn the Convergence Rates of Policy Gradient Methods
We consider infinite-horizon discounted Markov decision problems with finite state and action spaces and study the convergence rates of the projected policy gradient method and a general class of policy mirror descent me…
Policy Gradient Methods