SparseSwin: Swin Transformer with Sparse Transformer Block
Advancements in computer vision research have put transformer architecture as the state of the art in computer vision tasks. One of the known drawbacks of the transformer architecture is the high number of parameters, this can lead to a more complex and inefficient algorithm. This paper aims to reduce the number of parameters and in turn, made the transformer more efficient. We present Sparse Transformer (SparTa) Block, a modified transformer block with an addition of a sparse token converter that reduces the number of tokens used. We use the SparTa Block inside the Swin T architecture (SparseSwin) to leverage Swin capability to downsample its input and reduce the number of initial tokens to be calculated. The proposed SparseSwin model outperforms other state of the art models in image classification with an accuracy of 86.96%, 97.43%, and 85.35% on the ImageNet100, CIFAR10, and CIFAR100 datasets respectively. Despite its fewer parameters, the result highlights the potential of a transformer architecture using a sparse token converter with a limited number of tokens to optimize the use of the transformer and improve its performance.
Code (1)
Tasks
image-classificationImage ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SwInception -- Local Attention Meets Convolutions
Sparse vision transformers have gained popularity as efficient encoders for medical volumetric segmentation, with Swin emerging as a prominent choice. Swin uses local attention to reduce complexity and yields excellent p…
Image SegmentationHAAT: Hybrid Attention Aggregation Transformer for Image Super-Resolution
In the research area of image super-resolution, Swin-transformer-based models are favored for their global spatial modeling and shifting window attention mechanism. However, existing methods often limit self-attention to…
Image Super-ResolutionSuper-ResolutionResidual Swin Transformer Channel Attention Network for Image Demosaicing
Image demosaicing is problem of interpolating full- resolution color images from raw sensor (color filter array) data. During last decade, deep neural networks have been widely used in image restoration, and in particula…
DemosaickingImage RestorationSingle Image Super-Resolution Using Lightweight Networks Based on Swin Transformer
Image super-resolution reconstruction is an important task in the field of image processing technology, which can restore low resolution image to high quality image with high resolution. In recent years, deep learning ha…
Image Super-ResolutionSuper-ResolutionYuan-TecSwin: A text conditioned Diffusion model with Swin-transformer blocks
Diffusion models have shown remarkable capacity in image synthesis based on their U-shaped architecture and convolutional neural networks (CNN) as basic blocks. The locality of the convolution operation in CNN may limit …
Image Restoration