Is Large-Scale Pretraining the Secret to Good Domain Generalization?
Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains. Recent methods combine robust features from web-scale pretrained backbones with new features learned from source data, and this has dramatically improved benchmark results. However, it remains unclear if DG finetuning methods are becoming better over time, or if improved benchmark performance is simply an artifact of stronger pre-training. Prior studies have shown that perceptual similarity to pre-training data correlates with zero-shot performance, but we find the effect limited in the DG setting. Instead, we posit that having perceptually similar data in pretraining is not enough; and that it is how well these data were learned that determines performance. This leads us to introduce the Alignment Hypothesis, which states that the final DG performance will be high if and only if alignment of image and class label text embeddings is high. Our experiments confirm the Alignment Hypothesis is true, and we use it as an analysis tool of existing DG methods evaluated on DomainBed datasets by splitting evaluation data into In-pretraining (IP) and Out-of-pretraining (OOP). We show that all evaluated DG methods struggle on DomainBed-OOP, while recent methods excel on DomainBed-IP. Put together, our findings highlight the need for DG methods which can generalize beyond pretraining alignment.
Code (0)
등록된 구현이 없습니다.
Tasks
Domain GeneralizationSimilar Papers 제목 키워드 기반
Why does my medical AI look at pictures of birds? Exploring the efficacy of transfer learning across domain boundaries
It is an open secret that ImageNet is treated as the panacea of pretraining. Particularly in medical machine learning, models not trained from scratch are often finetuned based on ImageNet-pretrained models. We posit tha…
Computed Tomography (CT)Linear evaluationTransfer LearningNeural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
The goal of semi-supervised learning is to utilize the unlabeled, in-domain dataset U to improve models trained on the labeled dataset D. Under the context of large-scale language-model (LM) pretraining, how we can make …
General ClassificationLanguage ModellingPseudo Labeltext-classification+1Joint adjustment image steganography networks
Image steganography aims to achieve covert communication between two partners utilizing stego images generated by hiding secret images within cover images. Existing deep image steganography methods have been rapidly deve…
Image SteganographySteganographicsVideo Pretraining Advances 3D Deep Learning on Chest CT Tasks
Pretraining on large natural image classification datasets such as ImageNet has aided model development on data-scarce 2D medical tasks. 3D medical tasks often have much less data than 2D medical tasks, prompting practit…
Deep Learningimage-classificationImage ClassificationInterspecies evolutionary dynamics mediated by public goods in bacterial quorum sensing
Bacterial quorum sensing is the communication that takes place between bacteria as they secrete certain molecules into the intercellular medium that later get absorbed by the secreting cells themselves and by others. Dep…