Information Theory and its Relation to Machine Learning
In this position paper, I first describe a new perspective on machine learning (ML) by four basic problems (or levels), namely, "What to learn?", "How to learn?", "What to evaluate?", and "What to adjust?". The paper stresses more on the first level of "What to learn?", or "Learning Target Selection". Towards this primary problem within the four levels, I briefly review the existing studies about the connection between information theoretical learning (ITL [1]) and machine learning. A theorem is given on the relation between the empirically-defined similarity measure and information measures. Finally, a conjecture is proposed for pursuing a unified mathematical interpretation to learning target selection.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningPositionRelationSimilar Papers 제목 키워드 기반
Semi-bounded Rationality: A model for decision making
In this paper the theory of semi-bounded rationality is proposed as an extension of the theory of bounded rationality. In particular, it is proposed that a decision making process involves two components and these are th…
Decision MakingMissing ValuesmodelStructure Learning via Mutual Information
This paper presents a novel approach to machine learning algorithm design based on information theory, specifically mutual information (MI). We propose a framework for learning and representing functional relationships i…
Flexibly-bounded Rationality and Marginalization of Irrationality Theories for Decision Making
In this paper the theory of flexibly-bounded rationality which is an extension to the theory of bounded rationality is revisited. Rational decision making involves using information which is almost always imperfect and i…
Decision MakingA unified theory of information transfer and causal relation
Information transfer between coupled stochastic dynamics, measured by transfer entropy and information flow, is suggested as a physical process underlying the causal relation of systems. While information transfer analys…
Causal InferenceRelationMathematical Characterization of Signal Semantics and Rethinking of the Mathematical Theory of Information
Shannon information theory is established based on probability and bits, and the communication technology based on this theory realizes the information age. The original goal of Shannon's information theory is to describ…
Semantic Communication