Continual learning benefits from multiple sleep mechanisms: NREM, REM, and Synaptic Downscaling
Learning new tasks and skills in succession without losing prior learning (i.e., catastrophic forgetting) is a computational challenge for both artificial and biological neural networks, yet artificial systems struggle to achieve parity with their biological analogues. Mammalian brains employ numerous neural operations in support of continual learning during sleep. These are ripe for artificial adaptation. Here, we investigate how modeling three distinct components of mammalian sleep together affects continual learning in artificial neural networks: (1) a veridical memory replay process observed during non-rapid eye movement (NREM) sleep; (2) a generative memory replay process linked to REM sleep; and (3) a synaptic downscaling process which has been proposed to tune signal-to-noise ratios and support neural upkeep. We find benefits from the inclusion of all three sleep components when evaluating performance on a continual learning CIFAR-100 image classification benchmark. Maximum accuracy improved during training and catastrophic forgetting was reduced during later tasks. While some catastrophic forgetting persisted over the course of network training, higher levels of synaptic downscaling lead to better retention of early tasks and further facilitated the recovery of early task accuracy during subsequent training. One key takeaway is that there is a trade-off at hand when considering the level of synaptic downscaling to use - more aggressive downscaling better protects early tasks, but less downscaling enhances the ability to learn new tasks. Intermediate levels can strike a balance with the highest overall accuracies during training. Overall, our results both provide insight into how to adapt sleep components to enhance artificial continual learning systems and highlight areas for future neuroscientific sleep research to further such systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual Learningimage-classificationImage ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
NREM and REM: cognitive and energetic gains in thalamo-cortical sleeping and awake spiking model
Sleep is essential for learning and cognition, but the mechanisms by which it stabilizes learning, supports creativity, and manages the energy consumption of networks engaged in post-sleep task have not been yet modelled…
ManagementWake-Sleep Consolidated Learning
We propose Wake-Sleep Consolidated Learning (WSCL), a learning strategy leveraging Complementary Learning System theory and the wake-sleep phases of the human brain to improve the performance of deep neural networks for …
Continual LearningHippocampusSleepExplain: Explainable Non-Rapid Eye Movement and Rapid Eye Movement Sleep Stage Classification from EEG Signal
Classification of sleep stages is one of the most important diagnostic approaches for a variety of sleep-related disorders. Electroencephalography (EEG) is regarded as a powerful tool for examining the association betwee…
EEG Signal ClassificationAutomatic Micro-sleep Detection under Car-driving Simulation Environment using Night-sleep EEG
A micro-sleep is a short sleep that lasts from 1 to 30 secs. Its detection during driving is crucial to prevent accidents that could claim a lot of people's lives. Electroencephalogram (EEG) is suitable to detect micro-s…
EEGElectroencephalogram (EEG)Human Biometric Signals Monitoring based on WiFi Channel State Information using Deep Learning
In this paper, we first present a single-input, multiple-output convolutional neural network that can estimate both heart rate and respiration rate simultaneously by exploiting the underlying link between heart rate and …