Neural Entropy
We explore the connection between deep learning and information theory through the paradigm of diffusion models. A diffusion model converts noise into structured data by reinstating, imperfectly, information that is erased when data was diffused to noise. This information is stored in a neural network during training. We quantify this information by introducing a measure called neural entropy, which is related to the total entropy produced by diffusion. Neural entropy is a function of not just the data distribution, but also the diffusive process itself. Measurements of neural entropy on a few simple image diffusion models reveal that they are extremely efficient at compressing large ensembles of structured data.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Entropy, neutro-entropy and anti-entropy for neutrosophic information
This approach presents a multi-valued representation of the neutrosophic information. It highlights the link between the bifuzzy information and neutrosophic one. The constructed deca-valued structure shows the neutrosop…
Maximum Entropy of Random Permutation Set
Recently, a new type of set, named as random permutation set (RPS), is proposed by considering all the permutations of elements in a certain set. For measuring the uncertainty of RPS, the entropy of RPS is presented. How…
Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning ability of large language models. However, widely used RLVR algorithms, such as GRPO, often suffer from e…
Reinforcement LearningMathematical ReasoningOn Entropy Control in LLM-RL Algorithms
For RL algorithms, appropriate entropy control is crucial to their effectiveness. To control the policy entropy, a commonly used method is entropy regularization, which is adopted in various popular RL algorithms includi…
RNA thermodynamic structural entropy
Conformational entropy for atomic-level, three dimensional biomolecules is known experimentally to play an important role in protein-ligand discrimination, yet reliable computation of entropy remains a difficult problem.…