Contrastive Mutual Information Maximization for Binary Neural Networks
Neural network binarization accelerates deep models by quantizing their weights and activations into 1-bit. However, there is still a huge performance gap between Binary Neural Networks (BNNs) and their full-precision counterparts. As the quantization error caused by weights binarization has been reduced in earlier works, the activations binarization becomes the major obstacle for further improvement of the accuracy. In spite of studies about the full-precision networks highlighting the distributions of activations, few works study the distribution of the binary activations in BNNs. In this paper, we introduce mutual information as the metric to measure the information shared by the binary and the latent full-precision activations. Then we maximize the mutual information by establishing a contrastive learning framework while training BNNs. Specifically, the representation ability of the BNNs is greatly strengthened via pulling the positive pairs with binary and full-precision activations from the same input samples, as well as pushing negative pairs from different samples (the number of negative pairs can be exponentially large). This benefits the downstream tasks, not only classification but also segmentation and depth estimation, etc. The experimental results show that our method can be implemented as a pile-up module on existing state-of-the-art binarization methods and can remarkably improve the performance over them on CIFAR-10/100 and ImageNet, in addition to the good generalization ability on NYUD-v2.
Code (0)
등록된 구현이 없습니다.
Tasks
BinarizationContrastive LearningDepth EstimationQuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Cross-Domain Sentiment Classification with Contrastive Learning and Mutual Information Maximization
Contrastive learning (CL) has been successful as a powerful representation learning method. In this work we propose CLIM: Contrastive Learning with mutual Information Maximization, to explore the potential of CL on cross…
Contrastive LearningGeneral ClassificationRepresentation LearningSentiment Analysis+1Long-Tailed Recognition by Mutual Information Maximization between Latent Features and Ground-Truth Labels
Although contrastive learning methods have shown prevailing performance on a variety of representation learning tasks, they encounter difficulty when the training dataset is long-tailed. Many researchers have combined co…
Contrastive Learningimage-classificationImage ClassificationImage Segmentation+3Robust Contrastive Learning against Noisy Views
Contrastive learning relies on an assumption that positive pairs contain related views, e.g., patches of an image or co-occurring multimodal signals of a video, that share certain underlying information about an instance…
Binary ClassificationContrastive LearningA Mutual Information Perspective on Federated Contrastive Learning
We investigate contrastive learning in the federated setting through the lens of SimCLR and multi-view mutual information maximization. In doing so, we uncover a connection between contrastive representation learning and…
Contrastive LearningFederated Unsupervised LearningRepresentation LearningConstrained Multiview Representation for Self-supervised Contrastive Learning
Representation learning constitutes a pivotal cornerstone in contemporary deep learning paradigms, offering a conduit to elucidate distinctive features within the latent space and interpret the deep models. Nevertheless,…
Contrastive LearningDisentanglementImage SegmentationMedical Image Segmentation+3