Disentangled Information Bottleneck
The information bottleneck (IB) method is a technique for extracting information that is relevant for predicting the target random variable from the source random variable, which is typically implemented by optimizing the IB Lagrangian that balances the compression and prediction terms. However, the IB Lagrangian is hard to optimize, and multiple trials for tuning values of Lagrangian multiplier are required. Moreover, we show that the prediction performance strictly decreases as the compression gets stronger during optimizing the IB Lagrangian. In this paper, we implement the IB method from the perspective of supervised disentangling. Specifically, we introduce Disentangled Information Bottleneck (DisenIB) that is consistent on compressing source maximally without target prediction performance loss (maximum compression). Theoretical and experimental results demonstrate that our method is consistent on maximum compression, and performs well in terms of generalization, robustness to adversarial attack, out-of-distribution detection, and supervised disentangling.
Code (1)
Tasks
Adversarial AttackOut-of-Distribution DetectionPredictionSimilar Papers 제목 키워드 기반
IB-GAN: Disentangled Representation Learning with Information Bottleneck GAN
We present a novel architecture of GAN for a disentangled representation learning. The new model architecture is inspired by Information Bottleneck (IB) theory thereby named IB-GAN. IB-GAN objective is similar to that of…
DisentanglementRepresentation Learning$β$-CapsNet: Learning Disentangled Representation for CapsNet by Information Bottleneck
We present a framework for learning disentangled representation of CapsNet by information bottleneck constraint that distills information into a compact form and motivates to learn an interpretable factorized capsule. In…
DisentanglementVariational InferenceDisentangled Generation with Information Bottleneck for Few-Shot Learning
Few-shot learning (FSL), which aims to classify unseen classes with few samples, is challenging due to data scarcity. Although various generative methods have been explored for FSL, the entangled generation process of th…
DisentanglementDiversityFew-Shot LearningRepresentation LearningMultimodal Representation-disentangled Information Bottleneck for Multimodal Recommendation
Multimodal data has significantly advanced recommendation systems by integrating diverse information sources to model user preferences and item characteristics. However, these systems often struggle with redundant and ir…
Multimodal RecommendationRecommendation SystemsFAVAE: Sequence Disentanglement using Information Bottleneck Principle
We propose the factorized action variational autoencoder (FAVAE), a state-of-the-art generative model for learning disentangled and interpretable representations from sequential data via the information bottleneck withou…
DisentanglementRepresentation Learning