Enhancing Representations through Heterogeneous Self-Supervised Learning
Incorporating heterogeneous representations from different architectures has facilitated various vision tasks, e.g., some hybrid networks combine transformers and convolutions. However, complementarity between such heterogeneous architectures has not been well exploited in self-supervised learning. Thus, we propose Heterogeneous Self-Supervised Learning (HSSL), which enforces a base model to learn from an auxiliary head whose architecture is heterogeneous from the base model. In this process, HSSL endows the base model with new characteristics in a representation learning way without structural changes. To comprehensively understand the HSSL, we conduct experiments on various heterogeneous pairs containing a base model and an auxiliary head. We discover that the representation quality of the base model moves up as their architecture discrepancy grows. This observation motivates us to propose a search strategy that quickly determines the most suitable auxiliary head for a specific base model to learn and several simple but effective methods to enlarge the model discrepancy. The HSSL is compatible with various self-supervised methods, achieving superior performances on various downstream tasks, including image classification, semantic segmentation, instance segmentation, and object detection. Our source code will be made publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationInstance Segmentationobject-detectionObject DetectionRepresentation LearningSelf-Supervised LearningSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Cross-view Self-Supervised Learning on Heterogeneous Graph Neural Network via Bootstrapping
Heterogeneous graph neural networks can represent information of heterogeneous graphs with excellent ability. Recently, self-supervised learning manner is researched which learns the unique expression of a graph through …
Contrastive LearningGraph Neural NetworkSelf-Supervised LearningEnhancing 2D Representation Learning with a 3D Prior
Learning robust and effective representations of visual data is a fundamental task in computer vision. Traditionally, this is achieved by training models with labeled data which can be expensive to obtain. Self-supervise…
Representation LearningSelf-Supervised LearningGraph Masked Autoencoder for Spatio-Temporal Graph Learning
Effective spatio-temporal prediction frameworks play a crucial role in urban sensing applications, including traffic analysis, human mobility behavior modeling, and citywide crime prediction. However, the presence of dat…
Crime PredictionData AugmentationGraph LearningSelf-Supervised LearningHGOT: Self-supervised Heterogeneous Graph Neural Network with Optimal Transport
Heterogeneous Graph Neural Networks (HGNNs), have demonstrated excellent capabilities in processing heterogeneous information networks. Self-supervised learning on heterogeneous graphs, especially contrastive self-superv…
Graph Neural NetworkNode ClassificationSelf-Supervised LearningLearning Genomic Sequence Representations using Graph Neural Networks over De Bruijn Graphs
The rapid expansion of genomic sequence data calls for new methods to achieve robust sequence representations. Existing techniques often neglect intricate structural details, emphasizing mainly contextual information. To…
Contrastive LearningRetrieval