Connected Superlevel Set in (Deep) Reinforcement Learning and its Application to Minimax Theorems
The aim of this paper is to improve the understanding of the optimization landscape for policy optimization problems in reinforcement learning. Specifically, we show that the superlevel set of the objective function with respect to the policy parameter is always a connected set both in the tabular setting and under policies represented by a class of neural networks. In addition, we show that the optimization objective as a function of the policy parameter and reward satisfies a stronger "equiconnectedness" property. To our best knowledge, these are novel and previously unknown discoveries. We present an application of the connectedness of these superlevel sets to the derivation of minimax theorems for robust reinforcement learning. We show that any minimax optimization program which is convex on one side and is equiconnected on the other side observes the minimax equality (i.e. has a Nash equilibrium). We find that this exact structure is exhibited by an interesting robust reinforcement learning problem under an adversarial reward attack, and the validity of its minimax equality immediately follows. This is the first time such a result is established in the literature.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement Learningreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
Super Level Sets and Exponential Decay: A Synergistic Approach to Stable Neural Network Training
The objective of this paper is to enhance the optimization process for neural networks by developing a dynamic learning rate algorithm that effectively integrates exponential decay and advanced anti-overfitting strategie…
Information-Theoretic Minimax Regret Bounds for Reinforcement Learning based on Duality
We study agents acting in an unknown environment where the agent's goal is to find a robust policy. We consider robust policies as policies that achieve high cumulative rewards for all possible environments. To this end,…
Existence and Minimax Theorems for Adversarial Surrogate Risks in Binary Classification
Adversarial training is one of the most popular methods for training methods robust to adversarial attacks, however, it is not well-understood from a theoretical perspective. We prove and existence, regularity, and minim…
Adversarial RobustnessBinary ClassificationPolynomial Superlevel Set Representation of the Multistationarity Region of Chemical Reaction Networks
In this paper we introduce a new representation for the multistationarity region of a reaction network, using polynomial superlevel sets. The advantages of using this polynomial superlevel set representation over the alr…
Geometric Measurements of the Axiom of Choice in Neural Proof Embeddings
The axiom of choice has divided the foundations of mathematics for over a century, but the distinction between classical and constructive proofs has remained a philosophical and methodological one. We use Lean 4's kernel…