Cyclical Curriculum Learning

2022-02-11 · H. Toprak Kesgin, M. Fatih Amasyali

Artificial neural networks (ANN) are inspired by human learning. However, unlike human education, classical ANN does not use a curriculum. Curriculum Learning (CL) refers to the process of ANN training in which examples are used in a meaningful order. When using CL, training begins with a subset of the dataset and new samples are added throughout the training, or training begins with the entire dataset and the number of samples used is reduced. With these changes in training dataset size, better results can be obtained with curriculum, anti-curriculum, or random-curriculum methods than the vanilla method. However, a generally efficient CL method for various architectures and data sets is not found. In this paper, we propose cyclical curriculum learning (CCL), in which the data size used during training changes cyclically rather than simply increasing or decreasing. Instead of using only the vanilla method or only the curriculum method, using both methods cyclically like in CCL provides more successful results. We tested the method on 18 different data sets and 15 architectures in image and text classification tasks and obtained more successful results than no-CL and existing CL methods. We also have shown theoretically that it is less erroneous to apply CL and vanilla cyclically instead of using only CL or only vanilla method. The code of Cyclical Curriculum is available at https://github.com/CyclicalCurriculum/Cyclical-Curriculum.

📄 PDF Abstract BibTeX arXiv:2202.05531

Code (1)

cyclicalcurriculum/cyclical-curriculum 공식 구현 tf

Tasks

text-classification Text Classification

General Cyclical Training of Neural Networks

2022-02-17 · Leslie N. Smith

This paper describes the principle of "General Cyclical Training" in machine learning, where training starts and ends with "easy training" and the "hard training" happens during the middle epochs. We propose several mani…

Data Augmentation Knowledge Distillation

Deep Reinforcement Learning using Cyclical Learning Rates

2020-07-31 · Ralf Gulde, Marc Tuscher, Akos Csiszar, Oliver Riedel 외

Deep Reinforcement Learning (DRL) methods often rely on the meticulous tuning of hyperparameters to successfully resolve problems. One of the most influential parameters in optimization procedures based on stochastic gra…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

On Cyclical MCMC Sampling

2024-03-01 · LiWei Wang, Xinru Liu, Aaron Smith, Yves Atchade

Cyclical MCMC is a novel MCMC framework recently proposed by Zhang et al. (2019) to address the challenge posed by high-dimensional multimodal posterior distributions like those arising in deep learning. The algorithm wo…

Advancing NLP Models with Strategic Text Augmentation: A Comprehensive Study of Augmentation Methods and Curriculum Strategies

2024-02-14 · Himmet Toprak Kesgin, Mehmet Fatih Amasyali

This study conducts a thorough evaluation of text augmentation techniques across a variety of datasets and natural language processing (NLP) tasks to address the lack of reliable, generalized evidence for these methods. …

Sentiment Analysis Text Augmentation Topic Classification

Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

2019-02-11 · ICLR 2020 1 · Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen 외

The posteriors over neural network weights are high dimensional and multimodal. Each mode typically characterizes a meaningfully different representation of the data. We develop Cyclical Stochastic Gradient MCMC (SG-MCMC…

Bayesian Inference Deep Learning Stochastic Optimization