Relevant sparse codes with variational information bottleneck
In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximizes information about a 'relevance' variable, Y, while constraining the information encoded about the original data, X. Unfortunately however, the IB method is computationally demanding when data are high-dimensional and/or non-gaussian. Here we propose an approximate variational scheme for maximizing a lower bound on the IB objective, analogous to variational EM. Using this method, we derive an IB algorithm to recover features that are both relevant and sparse. Finally, we demonstrate how kernelized versions of the algorithm can be used to address a broad range of problems with non-linear relation between X and Y.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
There Was Never a Bottleneck in Concept Bottleneck Models
Deep learning representations are often difficult to interpret, which can hinder their deployment in sensitive applications. Concept Bottleneck Models (CBMs) have emerged as a promising approach to mitigate this issue by…
Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration
We propose a novel information bottleneck (IB) method named Drop-Bottleneck, which discretely drops features that are irrelevant to the target variable. Drop-Bottleneck not only enjoys a simple and tractable compression …
Adversarial RobustnessDimensionality ReductionRobust Information Bottleneck for Task-Oriented Communication with Digital Modulation
Task-oriented communications, mostly using learning-based joint source-channel coding (JSCC), aim to design a communication-efficient edge inference system by transmitting task-relevant information to the receiver. Howev…
InformativenessSelf-supervised Sequential Information Bottleneck for Robust Exploration in Deep Reinforcement Learning
Effective exploration is critical for reinforcement learning agents in environments with sparse rewards or high-dimensional state-action spaces. Recent works based on state-visitation counts, curiosity and entropy-maximi…
Deep Reinforcement LearningEfficient Explorationreinforcement-learningReinforcement Learning (RL)+2Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training
Information Bottleneck (IB) is a widely used framework that enables the extraction of information related to a target random variable from a source random variable. In the objective function, IB controls the trade-off be…
Data Compression