Energy--Information Trade-off Induces Continuous and Discontinuous Phase Transitions in Lateral Predictive Coding
Lateral predictive coding is a recurrent neural network which creates energy-efficient internal representations by exploiting statistical regularity in sensory inputs. Here we investigate the trade-off between information robustness and energy in a linear model of lateral predictive coding analytically and by numerical minimization of a free energy. We observe several phase transitions in the synaptic weight matrix, especially a continuous transition which breaks reciprocity and permutation symmetry and builds cyclic dominance and a discontinuous transition with the associated sudden emergence of tight balance between excitatory and inhibitory interactions. The optimal network follows an ideal-gas law in an extended temperature range and saturates the efficiency upper-bound of energy utilization. These results bring theoretical insights on the emergence and evolution of complex internal models in predictive processing systems.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Discontinuous phase transition of feature detection in lateral predictive coding
The brain adopts the strategy of lateral predictive coding (LPC) to construct optimal internal representations for salient features in input sensory signals to reduce the energetic cost of information transmission. Here …
Supertagging-based Parsing with Linear Context-free Rewriting Systems
We present the first supertagging-based parser for LCFRS. It utilizes neural classifiers and tremendously outperforms previous LCFRS-based parsers in both accuracy and parsing speed. Moreover, our results keep up with th…
AE-OT: A NEW GENERATIVE MODEL BASED ON EXTENDED SEMI-DISCRETE OPTIMAL TRANSPORT
Generative adversarial networks (GANs) have attracted huge attention due to its capability to generate visual realistic images. However, most of the existing models suffer from the mode collapse or mode mixture problems.…
DecoderGPUOptimal transport mapping via input convex neural networks
In this paper, we present a novel and principled approach to learn the optimal transport between two distributions, from samples. Guided by the optimal transport theory, we learn the optimal Kantorovich potential which i…
Regularity and Stability Properties of Selective SSMs with Discontinuous Gating
Deep Selective State-Space Models (SSMs), characterized by input-dependent, time-varying parameters, offer significant expressive power but pose challenges for stability analysis, especially with discontinuous gating sig…
State Space Models