Self-Supervised Representation Learning From Multi-Domain Data
We present an information-theoretically motivated constraint for self-supervised representation learning from multiple related domains. In contrast to previous self-supervised learning methods, our approach learns from multiple domains, which has the benefit of decreasing the build-in bias of individual domain, as well as leveraging information and allowing knowledge transfer across multiple domains. The proposed mutual information constraints encourage neural network to extract common invariant information across domains and to preserve peculiar information of each domain simultaneously. We adopt tractable upper and lower bounds of mutual information to make the proposed constraints solvable. The learned representation is more unbiased and robust toward the input images. Extensive experimental results on both multi-domain and large-scale datasets demonstrate the necessity and advantage of multi-domain self-supervised learning with mutual information constraints. Representations learned in our framework on state-of-the-art methods achieve improved performance than those learned on a single domain.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningSelf-Supervised LearningTransfer LearningSimilar Papers 제목 키워드 기반
Adapting Self-Supervised Representations to Multi-Domain Setups
Current state-of-the-art self-supervised approaches, are effective when trained on individual domains but show limited generalization on unseen domains. We observe that these models poorly generalize even when trained on…
DisentanglementRepresentation LearningImproving out-of-distribution generalization via multi-task self-supervised pretraining
Self-supervised feature representations have been shown to be useful for supervised classification, few-shot learning, and adversarial robustness. We show that features obtained using self-supervised learning are compara…
Adversarial RobustnessDomain GeneralizationFew-Shot LearningMulti-Task Learning+2Multi-Domain Self-Supervised Learning
Contrastive self-supervised learning has recently gained significant attention owing to its ability to learn improved feature representations without the use of label information. Current contrastive learning approaches,…
Contrastive LearningRepresentation LearningSelf-Supervised LearningSelf-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning
Self-supervised learning excels in learning representations from large amounts of unlabeled data, demonstrating success across multiple data modalities. Yet, extending self-supervised learning to new modalities is non-tr…
Molecular Property PredictionProperty PredictionSelf-Supervised LearningSelf-Supervised Learning of Remote Sensing Scene Representations Using Contrastive Multiview Coding
In recent years self-supervised learning has emerged as a promising candidate for unsupervised representation learning. In the visual domain its applications are mostly studied in the context of images of natural scenes.…
image-classificationImage ClassificationRemote Sensing Image ClassificationRepresentation Learning+1