Weight Freezing: A Regularization Approach for Fully Connected Layers with an Application in EEG Classification
In the realm of EEG decoding, enhancing the performance of artificial neural networks (ANNs) carries significant potential. This study introduces a novel approach, termed "weight freezing", that is anchored on the principles of ANN regularization and neuroscience prior knowledge. The concept of weight freezing revolves around the idea of reducing certain neurons' influence on the decision-making process for a specific EEG task by freezing specific weights in the fully connected layer during the backpropagation process. This is actualized through the use of a mask matrix and a threshold to determine the proportion of weights to be frozen during backpropagation. Moreover, by setting the masked weights to zero, weight freezing can not only realize sparse connections in networks with a fully connected layer as the classifier but also function as an efficacious regularization method for fully connected layers. Through experiments involving three distinct ANN architectures and three widely recognized EEG datasets, we validate the potency of weight freezing. Our method significantly surpasses previous peak performances in classification accuracy across all examined datasets. Supplementary control experiments offer insights into performance differences pre and post weight freezing implementation and scrutinize the influence of the threshold in the weight freezing process. Our study underscores the superior efficacy of weight freezing compared to traditional fully connected networks for EEG feature classification tasks. With its proven effectiveness, this innovative approach holds substantial promise for contributing to future strides in EEG decoding research.
Code (1)
Tasks
Decision MakingEEGEeg DecodingSimilar Papers 제목 키워드 기반
DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
Variants dropout methods have been designed for the fully-connected layer, convolutional layer and recurrent layer in neural networks, and shown to be effective to avoid overfitting. As an appealing alternative to recurr…
How transferable are features in convolutional neural network acoustic models across languages?
Characterization of the representations learned in intermediate layers of deep networks can provide valuable insight into the nature of a task and can guide the development of well-tailored learning strategies. Here we s…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Specificityspeech-recognition+2Visualization of fully connected layer weights in deep learning CT reconstruction
Recently, the use of deep learning techniques to reconstruct computed tomography (CT) images has become a hot research topic, including sinogram domain methods, image domain methods and sinogram domain to image domain me…
Computed Tomography (CT)CT ReconstructionAn Adaptive and Stability-Promoting Layerwise Training Approach for Sparse Deep Neural Network Architecture
This work presents a two-stage adaptive framework for progressively developing deep neural network (DNN) architectures that generalize well for a given training data set. In the first stage, a layerwise training approach…
Tensor Regression Networks with various Low-Rank Tensor Approximations
Tensor regression networks achieve high compression rate of neural networks while having slight impact on performances. They do so by imposing low tensor rank structure on the weight matrices of fully connected layers. I…
regression