Information Theoretic Interpretation of Deep learning
We interpret part of the experimental results of Shwartz-Ziv and Tishby [2017]. Inspired by these results, we established a conjecture of the dynamics of the machinary of deep neural network. This conjecture can be used to explain the counterpart result by Saxe et al. [2018].
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningSimilar Papers 제목 키워드 기반
Characterizing Tradeoffs in Language Model Decoding with Informational Interpretations
We propose a theoretical framework for formulating language model decoder algorithms with dynamic programming and information theory. With dynamic programming, we lift the design of decoder algorithms from the logit spac…
DecoderDiversityLanguage ModelingLanguage ModellingSemantic Geometry for policy-constrained interpretation
We present a geometric framework for policy-constrained semantic interpretation that provably prevents hallucinated commitments in high-stakes domains. Semantic meaning is represented as direction on a unit sphere, evide…
Bayesian InferenceA unified information-theoretic model of EEG signatures of human language processing
We advance an information-theoretic model of human language processing in the brain, in which incoming linguistic input is processed at two levels, in terms of a heuristic interpretation and in terms of error correction.…
EEGElectroencephalogram (EEG)ERPLearning to Explain: An Information-Theoretic Perspective on Model Interpretation
We introduce instancewise feature selection as a methodology for model interpretation. Our method is based on learning a function to extract a subset of features that are most informative for each given example. This fea…
feature selectionCommon Information Components Analysis
We give an information-theoretic interpretation of Canonical Correlation Analysis (CCA) via (relaxed) Wyner's common information. CCA permits to extract from two high-dimensional data sets low-dimensional descriptions (f…