Pattern Attention Transformer with Doughnut Kernel
We present in this paper a new architecture, the Pattern Attention Transformer (PAT), that is composed of the new doughnut kernel. Compared with tokens in the NLP field, Transformer in computer vision has the problem of handling the high resolution of pixels in images. In ViT, an image is cut into square-shaped patches. As the follow-up of ViT, Swin Transformer proposes an additional step of shifting to decrease the existence of fixed boundaries, which also incurs 'two connected Swin Transformer blocks' as the minimum unit of the model. Inheriting the patch/window idea, our doughnut kernel enhances the design of patches further. It replaces the line-cut boundaries with two types of areas: sensor and updating, which is based on the comprehension of self-attention (named QKVA grid). The doughnut kernel also brings a new topic about the shape of kernels beyond square. To verify its performance on image classification, PAT is designed with Transformer blocks of regular octagon shape doughnut kernels. Its architecture is lighter: the minimum pattern attention layer is only one for each stage. Under similar complexity of computation, its performances on ImageNet 1K reach higher throughput (+10%) and surpass Swin Transformer (+0.8 acc1).
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Modelling the Doughnut of social and planetary boundaries with frugal machine learning
The 'Doughnut' of social and planetary boundaries has emerged as a popular framework for assessing environmental and social sustainability. Here, we provide a proof-of-concept analysis that shows how machine learning (ML…
Reinforcement LearningHow rare are the properties of binary relations?
Knoblauch (2014) and Knoblauch (2015) investigate the relative size of the collection of binary relations with desirable features as compared to the set of all binary relations using symmetric difference metric (Cantor) …
Hilbert-Guided Sparse Local Attention
The quadratic compute and memory costs of global self-attention severely limit its use in high-resolution images. Local attention reduces complexity by restricting attention to neighborhoods. Block-sparse kernels can fur…
Impact of 3D UWB Antenna Radiation Pattern on Air-to-Ground Drone Connectivity
Three dimensional (3D) radiation pattern of an antenna mounted at a drone can significantly influence the air-to-ground (A2G) link quality. Even when a drone transmitter is very close to a ground receiver, if the antenna…
Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel
Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction. At the core of the Tran…
Machine TranslationTranslation