Memory Population in Continual Learning via Outlier Elimination

2022-07-04 · Julio Hurtado, Alain Raymond-Saez, Vladimir Araujo, Vincenzo Lomonaco, Alvaro Soto, Davide Bacciu

Catastrophic forgetting, the phenomenon of forgetting previously learned tasks when learning a new one, is a major hurdle in developing continual learning algorithms. A popular method to alleviate forgetting is to use a memory buffer, which stores a subset of previously learned task examples for use during training on new tasks. The de facto method of filling memory is by randomly selecting previous examples. However, this process could introduce outliers or noisy samples that could hurt the generalization of the model. This paper introduces Memory Outlier Elimination (MOE), a method for identifying and eliminating outliers in the memory buffer by choosing samples from label-homogeneous subpopulations. We show that a space with a high homogeneity is related to a feature space that is more representative of the class distribution. In practice, MOE removes a sample if it is surrounded by samples from different labels. We demonstrate the effectiveness of MOE on CIFAR-10, CIFAR-100, and CORe50, outperforming previous well-known memory population methods.

📄 PDF Abstract BibTeX arXiv:2207.01145

Code (1)

JuliousHurtado/MOE 공식 구현 pytorch

Tasks

Continual Learning

Methods 이 논문이 사용한 방법론

AWARE We propose to theoretically and empirically examine the effect of incorporating weighting schemes into walk-aggregating GNNs. To this end, we propose a simple, interpretable, and…

Experience Replay Experience Replay is a replay memory technique used in reinforcement learning where we store the agent’s experiences at each time-step, $e\_{t} = \left(s\_{t}, a\_{t}, r\_{t},…

Online Continual Learning from Imbalanced Data

2020-01-01 · ICML 2020 1 · Aristotelis Chrysakis, Marie-Francine Moens

A well-documented weakness of neural networks is the fact that they suffer from catastrophic forgetting when trained on data provided by a non-stationary distribution. Recent work in the field of continual learning attem…

Computational Efficiency Continual Learning

Information-theoretic Online Memory Selection for Continual Learning

2022-04-10 · ICLR 2022 4 · Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li 외

A challenging problem in task-free continual learning is the online selection of a representative replay memory from data streams. In this work, we investigate the online memory selection problem from an information-theo…

Continual Learning

Unsupervised Continual Learning and Self-Taught Associative Memory Hierarchies

2019-03-24 · ICLR Workshop LLD 2019 · James Smith, Seth Baer, Zsolt Kira, Constantine Dovrolis

We first pose the Unsupervised Continual Learning (UCL) problem: learning salient representations from a non-stationary stream of unlabeled data in which the number of object classes varies with time. Given limited label…

Continual Learning Online Clustering

TaskFusion: Continual Anomaly Detection for Heterogeneous Tabular Data

2026-06-10 · Dayananda Herurkar, Federico Raue, Joachim Folz, Jörn Hees 외 arxiv

Continual anomaly detection in tabular data is challenging and remains largely underexplored, particularly in settings with heterogeneous feature schemas, distribution shifts, and severe class imbalance. In many real-wor…

Continual Learning Anomaly Detection

RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization

2024-07-10 · Xijie Huang, Zechun Liu, Shih-Yang Liu, Kwang-Ting Cheng

Low-Rank Adaptation (LoRA), as a representative Parameter-Efficient Fine-Tuning (PEFT)method, significantly enhances the training efficiency by updating only a small portion of the weights in Large Language Models (LLMs)…

parameter-efficient fine-tuning Quantization