Boosting Network Weight Separability via Feed-Backward Reconstruction
This paper proposes a new evaluation metric and boosting method for weight separability in neural network design. In contrast to general visual recognition methods designed to encourage both intra-class compactness and inter-class separability of latent features, we focus on estimating linear independence of column vectors in weight matrix and improving the separability of weight vectors. To this end, we propose an evaluation metric for weight separability based on semi-orthogonality of a matrix and Frobenius distance, and the feed-backward reconstruction loss which explicitly encourages weight separability between the column vectors in the weight matrix. The experimental results on image classification and face recognition demonstrate that the weight separability boosting via minimization of feed-backward reconstruction loss can improve the visual recognition performance, hence universally boosting the performance on various visual recognition tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Face Recognitionimage-classificationImage ClassificationSimilar Papers 제목 키워드 기반
Learning the Connections in Direct Feedback Alignment
Feedback alignment was proposed to address the biological implausibility of the backpropagation algorithm which requires the transportation of the weight transpose during the backwards pass. The idea was later built upon…
Image ClassificationPREFER: Prompt Ensemble Learning via Feedback-Reflect-Refine
As an effective tool for eliciting the power of Large Language Models (LLMs), prompting has recently demonstrated unprecedented abilities across a variety of complex tasks. To further improve the performance, prompt ense…
Ensemble LearningHallucinationNeuralEQ: Neural-Network-Based Equalizer for High-Speed Wireline Communication
With the growing demand for high-bandwidth applications like video streaming and cloud services, the data transfer rates required for wireline communication keeps increasing, making the channel loss a major obstacle in a…
Spike-based causal inference for weight alignment
In artificial neural networks trained with gradient descent, the weights used for processing stimuli are also used during backward passes to calculate gradients. For the real brain to approximate gradients, gradient info…
Causal InferenceEconometricsFeedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form
In this paper we focus on the linear algebra theory behind feedforward (FNN) and recurrent (RNN) neural networks. We review backward propagation, including backward propagation through time (BPTT). Also, we obtain a new …
Form