Understanding Predictive Coding as an Adaptive Trust-Region Method
Predictive coding (PC) is a brain-inspired local learning algorithm that has recently been suggested to provide advantages over backpropagation (BP) in biologically relevant scenarios. While theoretical work has mainly focused on showing how PC can approximate BP in various limits, the putative benefits of "natural" PC are less understood. Here we develop a theory of PC as an adaptive trust-region (TR) algorithm that uses second-order information. We show that the learning dynamics of PC can be interpreted as interpolating between BP's loss gradient direction and a TR direction found by the PC inference dynamics. Our theory suggests that PC should escape saddle points faster than BP, a prediction which we prove in a shallow linear model and support with experiments on deeper networks. This work lays a foundation for understanding PC in deep and wide networks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Constraint Learning to Define Trust Regions in Predictive-Model Embedded Optimization
There is a recent proliferation of research on the integration of machine learning and optimization. One expansive area within this research stream is predictive-model embedded optimization, which proposes the use of pre…
Decision MakingInteractive $360^{\circ}$ Video Streaming Using FoV-Adaptive Coding with Temporal Prediction
For $360^{\circ}$ video streaming, FoV-adaptive coding that allocates more bits for the predicted user's field of view (FoV) is an effective way to maximize the rendered video quality under the limited bandwidth. We deve…
PredictionAdaptive Decoding via Hierarchical Neural Information Gradients in Mouse Visual Tasks
Understanding the encoding and decoding mechanisms of dynamic neural responses to different visual stimuli is an important topic in exploring how the brain represents visual information. Currently, hierarchically deep ne…
Multi-resolution intra-predictive coding of 3D point cloud attributes
We propose an intra frame predictive strategy for compression of 3D point cloud attributes. Our approach is integrated with the region adaptive graph Fourier transform (RAGFT), a multi-resolution transform formed by a co…
Region-specific Risk Quantification for Interpretable Prognosis of COVID-19
The COVID-19 pandemic has strained global public health, necessitating accurate diagnosis and intervention to control disease spread and reduce mortality rates. This paper introduces an interpretable deep survival predic…
COVID-19 DiagnosisDecision MakingPredictionPrognosis+2