Self-Pretraining for Small Datasets by Exploiting Patch Information
Deep learning tasks with small datasets are often tackled by pretraining models with large datasets on relevent tasks. Although pretraining methods mitigate the problem of overfitting, it can be difficult to find appropriate pretrained models sometimes. In this paper, we proposed a self-pretraininng method by exploiting patch information in the dataset itself without pretraining on other datasets. Our experiments show that the self-pretraining method leads to better performance than training from scratch both in the condition of not using other data.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Where are my Neighbors? Exploiting Patches Relations in Self-Supervised Vision Transformer
Vision Transformers (ViTs) enabled the use of the transformer architecture on vision tasks showing impressive performances when trained on big datasets. However, on relatively small datasets, ViTs are less accurate given…
Inductive BiasSelf-Supervised LearningAuxiliary Learning for Self-Supervised Video Representation via Similarity-based Knowledge Distillation
Despite the outstanding success of self-supervised pretraining methods for video representation learning, they generalise poorly when the unlabeled dataset for pretraining is small or the domain difference between unlabe…
Auxiliary LearningKnowledge DistillationRepresentation LearningPilotWiMAE: Pilot-Native Representation Learning for Wireless Channels
Channel foundation models assume access to fully observed channels, an assumption that fails in deployment. We introduce PilotWiMAE, a self-supervised framework whose encoder ingests noisy pilot observations directly and…
Representation LearningSlideCheck: Guiding Self-Supervised Pretraining of Pathology Foundation Models via Dataset Distributions
Pathology foundation models are pretrained on large streams of WSI-derived patches, while supervision during data construction is often slide-level, sparse, or heterogeneous. This mismatch makes it difficult to understan…
Selfie: Self-supervised Pretraining for Image Embedding
We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as…
Language ModelingLanguage ModellingMasked Language Modeling