paper-with-me

홈 › Papers

Phase Transitions for the Information Bottleneck in Representation Learning

2020-01-07 · ICLR 2020 1 · Tailin Wu, Ian Fischer

In the Information Bottleneck (IB), when tuning the relative strength between compression and prediction terms, how do the two terms behave, and what's their relationship with the dataset and the learned representation? In this paper, we set out to answer these questions by studying multiple phase transitions in the IB objective: $\text{IB}_\beta[p(z|x)] = I(X; Z) - \beta I(Y; Z)$ defined on the encoding distribution p(z|x) for input $X$, target $Y$ and representation $Z$, where sudden jumps of $dI(Y; Z)/d \beta$ and prediction accuracy are observed with increasing $\beta$. We introduce a definition for IB phase transitions as a qualitative change of the IB loss landscape, and show that the transitions correspond to the onset of learning new classes. Using second-order calculus of variations, we derive a formula that provides a practical condition for IB phase transitions, and draw its connection with the Fisher information matrix for parameterized models. We provide two perspectives to understand the formula, revealing that each IB phase transition is finding a component of maximum (nonlinear) correlation between $X$ and $Y$ orthogonal to the learned representation, in close analogy with canonical-correlation analysis (CCA) in linear settings. Based on the theory, we present an algorithm for discovering phase transition points. Finally, we verify that our theory and algorithm accurately predict phase transitions in categorical datasets, predict the onset of learning new classes and class difficulty in MNIST, and predict prominent phase transitions in CIFAR10.

📄 PDF Abstract BibTeX arXiv:2001.01878

Code (0)

등록된 구현이 없습니다.

Tasks

Representation Learning

Similar Papers 제목 키워드 기반

Deep Learning and the Information Bottleneck Principle

2015-03-09 · Naftali Tishby, Noga Zaslavsky

Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input…

Deep LearningGeneralization Bounds

Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning

2020-01-11 · Tailin Wu

How can we enable machines to make sense of the world, and become better at learning? To approach this goal, I believe viewing intelligence in terms of many integral aspects, and also a universal two-term tradeoff betwee…

BIG-bench Machine LearningCausal DiscoveryFew-Shot LearningRepresentation Learning+1

A Mechanism-Driven Theory of Phase Transitions in Active Learning

2026-06-30 · Julia Machnio, Mads Nielsen, Mostafa Mehdipour Ghazi arxiv

Active learning (AL) performance is known to be budget-dependent, yet regimes are typically defined by heuristic label counts that fail to generalize across datasets or architectures. We characterize AL dynamics by refra…

Active Learning

A Provably Convergent Information Bottleneck Solution via ADMM

2021-02-09 · Teng-Hui Huang, Aly El Gamal

The Information bottleneck (IB) method enables optimizing over the trade-off between compression of data and prediction accuracy of learned representations, and has successfully and robustly been applied to both supervis…

Information PlaneRepresentation Learning

Probing Criticality in Quantum Spin Chains with Neural Networks

2020-05-05 · A Berezutskii, M Beketov, D Yudin, Z Zimborás 외

The numerical emulation of quantum systems often requires an exponential number of degrees of freedom which translates to a computational bottleneck. Methods of machine learning have been used in adjacent fields for effe…

Dimensionality Reduction