Optimizing Information Bottleneck in Reinforcement Learning: A Stein Variational Approach

2021-01-01 · Pei Yingjun, Hou Xinwen, Li Jian, Lei Wang

The information bottleneck (IB) principle is an elegant and useful learning framework for extracting relevant information that an input feature contains about the target. The principle has been widely used in supervised and unsupervised learning. In this paper, we investigate the effectiveness of the IB framework in reinforcement learning (RL). We first derive the objective based on IB in reinforcement learning, then we analytically derive the optimal conditional distribution of the optimization problem. Following the variational information bottleneck (VIB), we provide a variational lower bound using a prior distribution. Unlike VIB, we propose to utilize the amortized Stein variational gradient method to optimize the lower bound. We incorporate this framework in two popular RL algorithms: the advantageous actor critic algorithm (A2C) and the proximal policy optimization algorithm (PPO). Our experimental results show that our framework can improve the sample efficiency of vanilla A2C and PPO. We also show that our method achieves better performance than VIB and mutual information neural estimation (MINE), two other popular approaches to optimize the information bottleneck framework in supervised learning.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

A2C A2C, or Advantage Actor Critic, is a synchronous version of the A3C policy gradient method. As an alternative to the asynchronous…

Learning Representations in Reinforcement Learning: an Information Bottleneck Approach

2020-01-01 · ICLR 2020 1 · Yingjun Pei, Xinwen Hou

The information bottleneck principle is an elegant and useful approach to representation learning. In this paper, we investigate the problem of representation learning in the context of reinforcement learning using the i…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Representation Learning

Learning Representations in Reinforcement Learning:An Information Bottleneck Approach

2019-11-12 · Pei Yingjun, Hou Xinwen

Reinforcement Learning Representation Learning

Bottlenecks CLUB: Unifying Information-Theoretic Trade-offs Among Complexity, Leakage, and Utility

2022-07-11 · Behrooz Razeghi, Flavio P. Calmon, Deniz Gunduz, Slava Voloshynovskiy

Bottleneck problems are an important class of optimization problems that have recently gained increasing attention in the domain of machine learning and information theory. They are widely used in generative models, fair…

Face Recognition Fairness Privacy Preserving Deep Learning Representation Learning+1

Bridging the Gap Between Variational Inference and Wasserstein Gradient Flows

2023-10-31 · Mingxuan Yi, Song Liu

Variational inference is a technique that approximates a target distribution by optimizing within the parameter space of variational families. On the other hand, Wasserstein gradient flows describe optimization within th…

Variational Inference

The Dual Information Bottleneck

2020-06-08 · Zoe Piran, Ravid Shwartz-Ziv, Naftali Tishby

The Information Bottleneck (IB) framework is a general characterization of optimal representations obtained using a principled approach for balancing accuracy and complexity. Here we present a new framework, the Dual Inf…

Information Plane