Improving Vision Transformers for Incremental Learning

2021-12-12 · Pei Yu, Yinpeng Chen, Ying Jin, Zicheng Liu

This paper proposes a working recipe of using Vision Transformer (ViT) in class incremental learning. Although this recipe only combines existing techniques, developing the combination is not trivial. Firstly, naive application of ViT to replace convolutional neural networks (CNNs) in incremental learning results in serious performance degradation. Secondly, we nail down three issues of naively using ViT: (a) ViT has very slow convergence when the number of classes is small, (b) more bias towards new classes is observed in ViT than CNN-based architectures, and (c) the conventional learning rate of ViT is too low to learn a good classifier layer. Finally, our solution, named ViTIL (ViT for Incremental Learning) achieves new state-of-the-art on both CIFAR and ImageNet datasets for all three class incremental learning setups by a clear margin. We believe this advances the knowledge of transformer in the incremental learning community. Code will be publicly released.

📄 PDF Abstract BibTeX arXiv:2112.06103

Code (0)

등록된 구현이 없습니다.

Tasks

class-incremental learning Class Incremental Learning Incremental Learning

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model

2023-05-18 · Patrick Kahardipraja, Brielen Madureira, David Schlangen

Language is by its very nature incremental in how it is produced and processed. This property can be exploited by NLP systems to produce fast responses, which has been shown to be beneficial for real-time interactive app…

Natural Language Understanding Sentence

Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners

2024-04-02 · CVPR 2024 1 · Keon-Hee Park, Kyungwoo Song, Gyeong-Moon Park

Few-Shot Class Incremental Learning (FSCIL) is a task that requires a model to learn new classes incrementally without forgetting when only a few samples for each class are given. FSCIL encounters two significant challen…

class-incremental learning Class Incremental Learning Few-Shot Class-Incremental Learning Incremental Learning+2

When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality

2024-02-20 · Brielen Madureira, Patrick Kahardipraja, David Schlangen

Incremental models that process sentences one token at a time will sometimes encounter points where more than one interpretation is possible. Causal models are forced to output one interpretation and continue, whereas mo…

Dependency Parsing

Preserving Locality in Vision Transformers for Class Incremental Learning

2023-04-14 · Bowen Zheng, Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan

Learning new classes without forgetting is crucial for real-world applications for a classification model. Vision Transformers (ViT) recently achieve remarkable performance in Class Incremental Learning (CIL). Previous w…