Contrastive Forward-Forward: A Training Algorithm of Vision Transformer

2025-02-01 · Hossein Aghagolzadeh, Mehdi Ezoji

Although backpropagation is widely accepted as a training algorithm for artificial neural networks, researchers are always looking for inspiration from the brain to find ways with potentially better performance. Forward-Forward is a new training algorithm that is more similar to what occurs in the brain, although there is a significant performance gap compared to backpropagation. In the Forward-Forward algorithm, the loss functions are placed after each layer, and the updating of a layer is done using two local forward passes and one local backward pass. Forward-Forward is in its early stages and has been designed and evaluated on simple multi-layer perceptron networks to solve image classification tasks. In this work, we have extended the use of this algorithm to a more complex and modern network, namely the Vision Transformer. Inspired by insights from contrastive learning, we have attempted to revise this algorithm, leading to the introduction of Contrastive Forward-Forward. Experimental results show that our proposed algorithm performs significantly better than the baseline Forward-Forward leading to an increase of up to 10% in accuracy and boosting the convergence speed by 5 to 20 times on Vision Transformer. Furthermore, if we take Cross Entropy as the baseline loss function in backpropagation, it will be demonstrated that the proposed modifications to the baseline Forward-Forward reduce its performance gap compared to backpropagation on Vision Transformer, and even outperforms it in certain conditions, such as inaccurate supervision.

📄 PDF Abstract BibTeX arXiv:2502.00571

Code (0)

등록된 구현이 없습니다.

Tasks

Contrastive Learning image-classification Image Classification

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Position-Wise Feed-Forward Layer 설명 없음

Adam 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Self-Contrastive Forward-Forward Algorithm

2024-09-17 · Xing Chen, Dongshu Liu, jeremie Laydevant, Julie Grollier

Agents that operate autonomously benefit from lifelong learning capabilities. However, compatible training algorithms must comply with the decentralized nature of these systems, which imposes constraints on both the para…

Contrastive Learning Lifelong learning

HCL-FF: Hierarchical and Contrastive Learning for Forward-Forward Algorithm

2026-05-24 · Jie-En Yao, Hong-En Chen, C. -C. Jay Kuo arxiv

Deep neural networks trained with backpropagation have achieved outstanding performance in vision tasks but remain biologically implausible, computationally demanding, and difficult to interpret. The Forward-Forward (FF)…

Contrastive Learning

Forward-Forward Contrastive Learning

2023-05-04 · Md. Atik Ahamed, Jin Chen, Abdullah-Al-Zubaer Imran

Medical image classification is one of the most important tasks for computer-aided diagnosis. Deep learning models, particularly convolutional neural networks, have been successfully used for disease classification from …

Classification Contrastive Learning image-classification Image Classification+1

FFR: Forward-Forward Learning for Regression

2026-06-02 · Xinyang Liu, Xuanyu Liang, Shiqi Ding, Boyang Li 외 arxiv

The Forward-Forward (FF) algorithm offers a computationally efficient and biologically plausible alternative to backpropagation (BP) by training neural networks through purely local, layer-wise optimization. However, FF …