FOCIL: Finetune-and-Freeze for Online Class Incremental Learning by Training Randomly Pruned Sparse Experts

2024-03-13 · Murat Onur Yildirim, Elif Ceren Gok Yildirim, Decebal Constantin Mocanu, Joaquin Vanschoren

Class incremental learning (CIL) in an online continual learning setting strives to acquire knowledge on a series of novel classes from a data stream, using each data point only once for training. This is more realistic compared to offline modes, where it is assumed that all data from novel class(es) is readily available. Current online CIL approaches store a subset of the previous data which creates heavy overhead costs in terms of both memory and computation, as well as privacy issues. In this paper, we propose a new online CIL approach called FOCIL. It fine-tunes the main architecture continually by training a randomly pruned sparse subnetwork for each task. Then, it freezes the trained connections to prevent forgetting. FOCIL also determines the sparsity level and learning rate per task adaptively and ensures (almost) zero forgetting across all tasks without storing any replay data. Experimental results on 10-Task CIFAR100, 20-Task CIFAR100, and 100-Task TinyImagenet, demonstrate that our method outperforms the SOTA by a large margin. The code is publicly available at https://github.com/muratonuryildirim/FOCIL.

📄 PDF Abstract BibTeX arXiv:2403.14684

Code (1)

muratonuryildirim/focil 공식 구현 pytorch

Tasks

class-incremental learning Class Incremental Learning Continual Learning Incremental Learning

PKI: Prior Knowledge-Infused Neural Network for Few-Shot Class-Incremental Learning

2026-01-13 · Kexin Baoa, Fanzhao Lin, Zichen Wang, Yong Li 외 arxiv

Few-shot class-incremental learning (FSCIL) aims to continually adapt a model on a limited number of new-class examples, facing two well-known challenges: catastrophic forgetting and overfitting to new classes. Existing …

Few-Shot Class-Incremental Learning

INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Models

2025-11-24 · Parsa Madinei, Ryan Solgi, Ziqi Wen, Jonathan Skaza 외 arxiv

We introduce INTERLACE, a novel framework that prunes redundant layers in VLMs while maintaining performance through sample-efficient finetuning. Existing layer pruning methods lead to significant performance drop when a…

In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization

2024-04-25 · Herilalaina Rakotoarison, Steven Adriaensen, Neeratyoy Mallik, Samir Garibov 외

With the increasing computational costs associated with deep learning, automated hyperparameter optimization methods, strongly relying on black-box Bayesian optimization (BO), face limitations. Freeze-thaw BO offers a pr…

Bayesian Optimization Hyperparameter Optimization In-Context Learning

Don't freeze: Finetune encoders for better Self-Supervised HAR

2023-07-03 · Vitor Fortes Rey, Dominique Nshimyimana, Paul Lukowicz

Recently self-supervised learning has been proposed in the field of human activity recognition as a solution to the labelled data availability problem. The idea being that by using pretext tasks such as reconstruction or…

Activity Recognition Human Activity Recognition Self-Supervised Learning

Bias Mitigating Few-Shot Class-Incremental Learning

2024-02-01 · Li-Jun Zhao, Zhen-Duo Chen, Zi-Chao Zhang, Xin Luo 외

Few-shot class-incremental learning (FSCIL) aims at recognizing novel classes continually with limited novel class samples. A mainstream baseline for FSCIL is first to train the whole model in the base session, then free…

class-incremental learning Class Incremental Learning Few-Shot Class-Incremental Learning Incremental Learning