The Multi-layer Information Bottleneck Problem
The muti-layer information bottleneck (IB) problem, where information is propagated (or successively refined) from layer to layer, is considered. Based on information forwarded by the preceding layer, each stage of the network is required to preserve a certain level of relevance with regards to a specific hidden variable, quantified by the mutual information. The hidden variables and the source can be arbitrarily correlated. The optimal trade-off between rates of relevance and compression (or complexity) is obtained through a single-letter characterization, referred to as the rate-relevance region. Conditions of successive refinabilty are given. Binary source with BSC hidden variables and binary source with BSC/BEC mixed hidden variables are both proved to be successively refinable. We further extend our result to Guassian models. A counterexample of successive refinability is also provided.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Neural Network Activation Quantization with Bitwise Information Bottlenecks
Recent researches on information bottleneck shed new light on the continuous attempts to open the black box of neural signal encoding. Inspired by the problem of lossy signal compression for wireless communication, this …
Computational EfficiencyQuantizationComprehensive Information Bottleneck for Unveiling Universal Attribution to Interpret Vision Transformers
The feature attribution method reveals the contribution of input variables to the decision-making process to provide an attribution map for explanation. Existing methods grounded on the information bottleneck principle c…
Comprehensive Information Bottleneck for Unveiling Universal Attribution to Interpret Vision Transformers
The feature attribution method reveals the contribution of input variables to the decision-making process to provide an attribution map for explanation. Existing methods grounded on the information bottleneck princip…
Decision MakingLayer-wise Learning of Stochastic Neural Networks with Information Bottleneck
Information Bottleneck (IB) is a generalization of rate-distortion theory that naturally incorporates compression and relevance trade-offs for learning. Though the original IB has been extensively studied, there has not …
Adversarial RobustnessCaveats for information bottleneck in deterministic scenarios
Information bottleneck (IB) is a method for extracting information from one random variable $X$ that is relevant for predicting another random variable $Y$. To do so, IB identifies an intermediate "bottleneck" variable $…