DPT: Deformable Patch-based Transformer for Visual Recognition
Transformer has achieved great success in computer vision, while how to split patches in an image remains a problem. Existing methods usually use a fixed-size patch embedding which might destroy the semantics of objects. To address this problem, we propose a new Deformable Patch (DePatch) module which learns to adaptively split the images into patches with different positions and scales in a data-driven way rather than using predefined fixed patches. In this way, our method can well preserve the semantics in patches. The DePatch module can work as a plug-and-play module, which can easily be incorporated into different transformers to achieve an end-to-end training. We term this DePatch-embedded transformer as Deformable Patch-based Transformer (DPT) and conduct extensive evaluations of DPT on image classification and object detection. Results show DPT can achieve 81.9% top-1 accuracy on ImageNet classification, and 43.7% box mAP with RetinaNet, 44.3% with Mask R-CNN on MSCOCO object detection. Code has been made available at: https://github.com/CASIA-IVA-Lab/DPT .
Code (1)
Tasks
image-classificationImage Classificationobject-detectionObject DetectionSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Cross-Modal Learning with 3D Deformable Attention for Action Recognition
An important challenge in vision-based action recognition is the embedding of spatiotemporal features with two or more heterogeneous modalities into a single feature. In this study, we propose a new 3D deformable transfo…
Action RecognitionAttention Deficit is Ordered! Fooling Deformable Vision Transformers with Collaborative Adversarial Patches
The latest generation of transformer-based vision models has proven to be superior to Convolutional Neural Network (CNN)-based models across several vision tasks, largely attributed to their remarkable prowess in relatio…
object-detectionObject Detection3D Transformer based on deformable patch location for differential diagnosis between Alzheimer's disease and Frontotemporal dementia
Alzheimer's disease and Frontotemporal dementia are common types of neurodegenerative disorders that present overlapping clinical symptoms, making their differential diagnosis very challenging. Numerous efforts have been…
Data AugmentationDiagnosticPlaceFormer: Transformer-based Visual Place Recognition using Multi-Scale Patch Selection and Fusion
Visual place recognition is a challenging task in the field of computer vision, and autonomous robotics and vehicles, which aims to identify a location or a place from visual inputs. Contemporary methods in visual place …
Computational EfficiencyImage RetrievalVisual Place RecognitionDAT++: Spatially Dynamic Vision Transformer with Deformable Attention
Transformers have shown superior performance on various vision tasks. Their large receptive field endows Transformer models with higher representation power than their CNN counterparts. Nevertheless, simply enlarging the…
Image ClassificationInstance SegmentationObject DetectionSemantic Segmentation