Exploring Modulated Detection Transformer as a Tool for Action Recognition in Videos
During recent years transformers architectures have been growing in popularity. Modulated Detection Transformer (MDETR) is an end-to-end multi-modal understanding model that performs tasks such as phase grounding, referring expression comprehension, referring expression segmentation, and visual question answering. One remarkable aspect of the model is the capacity to infer over classes that it was not previously trained for. In this work we explore the use of MDETR in a new task, action detection, without any previous training. We obtain quantitative results using the Atomic Visual Actions dataset. Although the model does not report the best performance in the task, we believe that it is an interesting finding. We show that it is possible to use a multi-modal model to tackle a task that it was not designed for. Finally, we believe that this line of research may lead into the generalization of MDETR in additional downstream tasks.
Code (1)
Tasks
Action DetectionAction RecognitionAction Recognition In VideosQuestion AnsweringReferring ExpressionReferring Expression ComprehensionReferring Expression SegmentationVisual Question AnsweringVisual Question Answering (VQA)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Dynamic Query Combinations for Transformer-based Object Detection and Segmentation
Transformer-based detection and segmentation methods use a list of learned detection queries to retrieve information from the transformer network and learn to predict the location and category of one specific object from…
Instance SegmentationObjectobject-detectionObject Detection+4QUEST: A robust attention formulation using query-modulated spherical attention
The Transformer model architecture has become one of the most widely used in deep learning and the attention mechanism is at its core. The standard attention formulation uses a softmax operation applied to a scaled dot p…
Band-Attention Modulated RetNet for Face Forgery Detection
The transformer networks are extensively utilized in face forgery detection due to their scalability across large datasets.Despite their success, transformers face challenges in balancing the capture of global context, w…
Fast Convergence of DETR with Spatially Modulated Co-Attention
The recently proposed Detection Transformer (DETR) model successfully applies Transformer to objects detection and achieves comparable performance with two-stage object detection frameworks, such as Faster-RCNN. However,…
Decoderobject-detectionObject DetectionFast Convergence of DETR with Spatially Modulated Co-Attention
The recently proposed Detection Transformer (DETR) model successfully applies Transformer to objects detection and achieves comparable performance with two-stage object detection frameworks, such as Faster-RCNN. However,…
Decoderobject-detectionObject Detection