Retina Vision Transformer (RetinaViT): Introducing Scaled Patches into Vision Transformers
Humans see low and high spatial frequency components at the same time, and combine the information from both to form a visual scene. Drawing on this neuroscientific inspiration, we propose an altered Vision Transformer architecture where patches from scaled down versions of the input image are added to the input of the first Transformer Encoder layer. We name this model Retina Vision Transformer (RetinaViT) due to its inspiration from the human visual system. Our experiments show that when trained on the ImageNet-1K dataset with a moderate configuration, RetinaViT achieves a 3.3% performance improvement over the original ViT. We hypothesize that this improvement can be attributed to the inclusion of low spatial frequency components in the input, which improves the ability to capture structural features, and to select and forward important features to deeper layers. RetinaViT thereby opens doors to further investigations into vertical pathways and attention patterns.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Pyramid U-Net for Retinal Vessel Segmentation
Retinal blood vessel can assist doctors in diagnosis of eye-related diseases such as diabetes and hypertension, and its segmentation is particularly important for automatic retinal image analysis. However, it is challeng…
DecoderRetinal Vessel SegmentationSegmentationMoCoViT: Mobile Convolutional Vision Transformer
Recently, Transformer networks have achieved impressive results on a variety of vision tasks. However, most of them are computationally expensive and not suitable for real-world mobile applications. In this work, we pres…
object-detectionObject DetectionBenchmarking Convolutional, Transformer, Hybrid, and Vision Language Models for Multi Disease Retinal Screening
Modern deep learning offers powerful tools for automated retinal screening, but it remains unclear how different visual model families compare in realistic multi-disease settings and under domain shift. In this work, we …
Multi-Label ClassificationVTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers
In Fluorescein Angiography (FA), an exogenous dye is injected in the bloodstream to image the vascular structure of the retina. The injected dye can cause adverse reactions such as nausea, vomiting, anaphylactic shock, a…
Disease PredictionFundus to Angiography GenerationGenerative Adversarial NetworkImage GenerationTSPTQ-ViT: Two-scaled post-training quantization for vision transformer
Vision transformers (ViTs) have achieved remarkable performance in various computer vision tasks. However, intensive memory and computation requirements impede ViTs from running on resource-constrained edge devices. Due …
Quantization