Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization
"Forward-only" algorithms, which train neural networks while avoiding a backward pass, have recently gained attention as a way of solving the biologically unrealistic aspects of backpropagation. Here, we first address compelling challenges related to the "forward-only" rules, which include reducing the performance gap with backpropagation and providing an analytical understanding of their dynamics. To this end, we show that the forward-only algorithm with top-down feedback is well-approximated by an "adaptive-feedback-alignment" algorithm, and we analytically track its performance during learning in a prototype high-dimensional setting. Then, we compare different versions of forward-only algorithms, focusing on the Forward-Forward and PEPITA frameworks, and we show that they share the same learning principles. Overall, our work unveils the connections between three key neuro-inspired learning rules, providing a link between "forward-only" algorithms, i.e., Forward-Forward and PEPITA, and an approximation of backpropagation, i.e., Feedback Alignment.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Spin Glass Characterization of Neural Networks
This work presents a statistical mechanics characterization of neural networks, motivated by the replica symmetry breaking (RSB) phenomenon in spin glasses. A Hopfield-type spin glass model is constructed from a given fe…
Benchmarking Empirical and Learning-Based Approaches for Feedforward Steering Control in Autonomous Racing
Feedforward steering control is a key component of hierarchical control architectures for autonomous racing. The goal is to reduce steering corrections from the feedback controllers by predicting the vehicle's inverse la…
Trajectory PlanningSteering ControlSensor Placement on a Cantilever Beam Using Observability Gramians
Working from an observability characterization based on output energy sensitivity to changes in initial conditions, we derive both analytical and empirical observability Gramian tools for a class of continuum material sy…
Cantilever BeamSensitivityA Sensory Feedback Control Law for Octopus Arm Movements
The main contribution of this paper is a novel sensory feedback control law for an octopus arm. The control law is inspired by, and helps integrate, several observations made by biologists. The proposed control law is di…
Complete mathematical characterization of two simple toggle-switch biological systems
Prokaryotic gene expression is dynamic and noisy, and can lead to phenotypic variations. Previous work has shown that the selection of such variation can be modeled through basic reaction networks described by O.D.E. dis…
Vocal Bursts Valence Prediction