A Comprehensive Study of Vision Transformers on Dense Prediction Tasks
Convolutional Neural Networks (CNNs), architectures consisting of convolutional layers, have been the standard choice in vision tasks. Recent studies have shown that Vision Transformers (VTs), architectures based on self-attention modules, achieve comparable performance in challenging tasks such as object detection and semantic segmentation. However, the image processing mechanism of VTs is different from that of conventional CNNs. This poses several questions about their generalizability, robustness, reliability, and texture bias when used to extract features for complex tasks. To address these questions, we study and compare VT and CNN architectures as feature extractors in object detection and semantic segmentation. Our extensive empirical results show that the features generated by VTs are more robust to distribution shifts, natural corruptions, and adversarial attacks in both tasks, whereas CNNs perform better at higher image resolutions in object detection. Furthermore, our results demonstrate that VTs in dense prediction tasks produce more reliable and less texture-biased predictions.
Code (0)
등록된 구현이 없습니다.
Tasks
Objectobject-detectionObject DetectionSegmentationSemantic SegmentationSimilar Papers 제목 키워드 기반
Vision Transformers for Dense Prediction
We introduce dense vision transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision…
DecoderDepth EstimationMonocular Depth EstimationPrediction+1Full-resolution MLPs Empower Medical Dense Prediction
Dense prediction is a fundamental requirement for many medical vision tasks such as medical image restoration, registration, and segmentation. The most popular vision model, Convolutional Neural Networks (CNNs), has reac…
AnatomyImage RestorationPredictionSelf-Supervised Pre-training of Vision Transformers for Dense Prediction Tasks
We present a new self-supervised pre-training of Vision Transformers for dense prediction tasks. It is based on a contrastive loss across views that compares pixel-level representations to global image representations. T…
Depth EstimationMonocular Depth EstimationPredictionSemantic SegmentationFair Comparison between Efficient Attentions
Transformers have been successfully used in various fields and are becoming the standard tools in computer vision. However, self-attention, a core component of transformers, has a quadratic complexity problem, which limi…
Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning
Vision transformers have recently achieved competitive results across various vision tasks but still suffer from heavy computation costs when processing a large number of tokens. Many advanced approaches have been develo…
ClusteringDepth EstimationGPUimage-classification+8