The Forward-Forward Algorithm: Characterizing Training Behavior
The Forward-Forward algorithm is an alternative learning method which consists of two forward passes rather than a forward and backward pass employed by backpropagation. Forward-Forward networks employ layer local loss functions which are optimized based on the layer activation for each forward pass rather than a single global objective function. This work explores the dynamics of model and layer accuracy changes in Forward-Forward networks as training progresses in pursuit of a mechanistic understanding of their internal behavior. Treatments to various system characteristics are applied to investigate changes in layer and overall model accuracy as training progresses, how accuracy is impacted by layer depth, and how strongly individual layer accuracy is correlated with overall model accuracy. The empirical results presented suggest that layers deeper within Forward-Forward networks experience a delay in accuracy improvement relative to shallower layers and that shallower layer accuracy is strongly correlated with overall model accuracy.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
On the Convergence of Stochastic Gradient Descent with Perturbed Forward-Backward Passes
We study stochastic gradient descent (SGD) for composite optimization problems with $N$ sequential operators subject to perturbations in both the forward and backward passes. Unlike classical analyses that treat gradient…
Feed-forward Neural Networks with Trainable Delay
In this paper we build a bridge between feed-forward neural networks and delayed dynamical systems. As an initial demonstration, we capture the car-following behavior of a connected automated vehicle including time delay…
Approximating two value functions instead of one: towards characterizing a new family of Deep Reinforcement Learning algorithms
This paper makes one step forward towards characterizing a new family of \textit{model-free} Deep Reinforcement Learning (DRL) algorithms. The aim of these algorithms is to jointly learn an approximation of the state-val…
Deep Reinforcement LearningQ-LearningReinforcement LearningRelational Forward Models for Multi-Agent Learning
The behavioral dynamics of multi-agent systems have a rich and orderly structure, which can be leveraged to understand these systems, and to improve how artificial agents learn to operate in them. Here we introduce Relat…
Quantifying training challenges of dependency parsers
Not all dependencies are equal when training a dependency parser: some are straightforward enough to be learned with only a sample of data, others embed more complexity. This work introduces a series of metrics to quanti…
Cross-Lingual TransferDependency Parsing