Information theoretic learning of robust deep representations
We propose a novel objective function for learning robust deep representations of data based on information theory. Data is projected into a feature-vector space such that the mutual information of all subsets of features relative to the supervising signal is maximized. This objective function gives rise to robust representations by conserving available information relative to supervision in the face of noisy or unavailable features. Although the objective function is not directly tractable, we are able to derive a surrogate objective function. Minimizing this surrogate loss encourages features to be non-redundant and conditionally independent relative to the supervising signal. To evaluate the quality of obtained solutions, we have performed a set of preliminary experiments that show promising results.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
INFORMATION MAXIMIZATION AUTO-ENCODING
We propose the Information Maximization Autoencoder (IMAE), an information theoretic approach to simultaneously learn continuous and discrete representations in an unsupervised setting. Unlike the Variational Autoencoder…
DecoderDisentanglementInformativenessvalidBird's Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approach
NLP has a rich history of representing our prior understanding of language in the form of graphs. Recent work on analyzing contextualized text representations has focused on hand-designed probe models to understand how a…
Towards Comprehensive Information-theoretic Multi-view Learning
Information theory has inspired numerous advancements in multi-view learning. Most multi-view methods incorporating information-theoretic principles rely an assumption called multi-view redundancy which states that commo…
Representation LearningWhich Mutual-Information Representation Learning Objectives are Sufficient for Control?
Mutual information maximization provides an appealing formalism for learning representations of data. In the context of reinforcement learning (RL), such representations can accelerate learning by discarding irrelevant a…
Reinforcement Learning (RL)Representation LearningInformation Theory-Guided Heuristic Progressive Multi-View Coding
Multi-view representation learning captures comprehensive information from multiple views of a shared context. Recent works intuitively apply contrastive learning (CL) to learn representations, regarded as a pairwise man…
Contrastive LearningMULTI-VIEW LEARNINGRepresentation Learning