A Practical & Unified Notation for Information-Theoretic Quantities in ML
A practical notation can convey valuable intuitions and concisely express new ideas. Information theory is of importance to machine learning, but the notation for information-theoretic quantities is sometimes opaque. We propose a practical and unified notation and extend it to include information-theoretic quantities between observed outcomes (events) and random variables. This includes the point-wise mutual information known in NLP and mixed quantities such as specific surprise and specific information in the cognitive sciences and information gain in Bayesian optimal experimental design. We apply our notation to prove a version of Stirling's approximation for binomial coefficients mentioned by MacKa (2003) using new intuitions. We also concisely rederive the evidence lower bound for variational auto-encoders and variational inference in approximate Bayesian neural networks. Furthermore, we apply the notation to a popular information-theoretic acquisition function in Bayesian active learning which selects the most informative (unlabelled) samples to be labelled by an expert and extend this acquisition function to the core-set problem with the goal of selecting the most informative samples given the labels.
Code (0)
등록된 구현이 없습니다.
Tasks
Active LearningExperimental DesignVariational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitions
At its core, this thesis aims to enhance the practicality of deep learning by improving the label and training efficiency of deep learning models. To this end, we investigate data subset selection techniques, specificall…
Active LearningDeep LearningUnified Optimization of Source Weights and Transfer Quantities in Multi-Source Transfer Learning: An Asymptotic Framework
In multi-source transfer learning, a key challenge lies in how to appropriately differentiate and utilize heterogeneous source tasks. However, existing multi-source methods typically focus on optimizing either the source…
Multi-Task LearningTransfer LearningOrthogonal Representation Learning for Estimating Causal Quantities
Representation learning is widely used for estimating causal quantities (e.g., the conditional average treatment effect) from observational data. While existing representation learning methods have the benefit of allowin…
Representation LearningUnifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantities
Recently proposed methods in data subset selection, that is active learning and active sampling, use Fisher information, Hessians, similarity matrices based on gradients, and gradient lengths to estimate how informative …
Active LearningInformativenessOne if by Land, Two if by Sea, Three if by Four Seas, and More to Come -- Values of Perception, Prediction, Communication, and Common Sense in Decision Making
This work aims to rigorously define the values of perception, prediction, communication, and common sense in decision making. The defined quantities are decision-theoretic, but have information-theoretic analogues, e.g.,…
Decision Making