Configuring Data Augmentations to Reduce Variance Shift in Positional Embedding of Vision Transformers
Vision transformers (ViTs) have demonstrated remarkable performance in a variety of vision tasks. Despite their promising capabilities, training a ViT requires a large amount of diverse data. Several studies empirically found that using rich data augmentations, such as Mixup, Cutmix, and random erasing, is critical to the successful training of ViTs. Now, the use of rich data augmentations has become a standard practice in the current state. However, we report a vulnerability to this practice: Certain data augmentations such as Mixup cause a variance shift in the positional embedding of ViT, which has been a hidden factor that degrades the performance of ViT during the test phase. We claim that achieving a stable effect from positional embedding requires a specific condition on the image, which is often broken for the current data augmentation methods. We provide a detailed analysis of this problem as well as the correct configuration for these data augmentations to remove the side effects of variance shift. Experiments showed that adopting our guidelines improves the performance of ViTs compared with the current configuration of data augmentations.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Steerable Equivariant Representation Learning
Pre-trained deep image representations are useful for post-training tasks such as classification through transfer learning, image retrieval, and object detection. Data augmentations are a crucial aspect of pre-training r…
Image Retrievalobject-detectionObject DetectionOut-of-Distribution Detection+3Is Robustness Robust? On the interaction between augmentations and corruptions
Invariance to a broad array of image corruptions, such as warping, noise, or color shifts, is an important aspect of building robust models in computer vision. Recently, several new data augmentations have been proposed …
On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness
Invariance to a broad array of image corruptions, such as warping, noise, or color shifts, is an important aspect of building robust models in computer vision. Recently, several new data augmentations have been proposed …
Adversarial Label Invariant Graph Data Augmentations for Out-of-Distribution Generalization
Out-of-distribution (OoD) generalization occurs when representation learning encounters a distribution shift. This occurs frequently in practice when training and testing data come from different environments. Covariate …
Representation LearningGraph ClassificationObject-aware Contrastive Learning for Debiased Scene Representation
Contrastive self-supervised learning has shown impressive results in learning visual representations from unlabeled images by enforcing invariance against different data augmentations. However, the learned representation…
Contrastive LearningObjectRepresentation LearningSelf-Supervised Learning