Generalizing Information to the Evolution of Rational Belief
Information theory provides a mathematical foundation to measure uncertainty in belief. Belief is represented by a probability distribution that captures our understanding of an outcome's plausibility. Information measures based on Shannon's concept of entropy include realization information, Kullback-Leibler divergence, Lindley's information in experiment, cross entropy, and mutual information. We derive a general theory of information from first principles that accounts for evolving belief and recovers all of these measures. Rather than simply gauging uncertainty, information is understood in this theory to measure change in belief. We may then regard entropy as the information we expect to gain upon realization of a discrete latent random variable. This theory of information is compatible with the Bayesian paradigm in which rational belief is updated as evidence becomes available. Furthermore, this theory admits novel measures of information with well-defined properties, which we explore in both analysis and experiment. This view of information illuminates the study of machine learning by allowing us to quantify information captured by a predictive model and distinguish it from residual information contained in training data. We gain related insights regarding feature selection, anomaly detection, and novel Bayesian approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Anomaly Detectionfeature selectionSimilar Papers 제목 키워드 기반
A maximum entropy model of bounded rational decision-making with prior beliefs and market feedback
Bounded rationality is an important consideration stemming from the fact that agents often have limits on their processing abilities, making the assumption of perfect rationality inapplicable to many real tasks. We propo…
Decision MakingExtending rational models of communication from beliefs to actions
Speakers communicate to influence their partner's beliefs and shape their actions. Belief- and action-based objectives have been explored independently in recent computational models, but it has been challenging to expli…
Characterizing Assumption of Rationality by Incomplete Information
We characterize common assumption of rationality of 2-person games within an incomplete information framework. We use the lexicographic model with incomplete information and show that a belief hierarchy expresses common …
Iterated Belief Change Due to Actions and Observations
In action domains where agents may have erroneous beliefs, reasoning about the effects of actions involves reasoning about belief change. In this paper, we use a transition system approach to reason about the evolution o…
Self-selection of Information and Belief Update: An Experiment on COVID-19 Vaccine Information Acquisition
Rational information acquisition theory predicts that people select the more informative information. Thus, people's beliefs will be more persuaded by the information they select. We test the prediction in a critical rea…