Lite-MDETR: A Lightweight Multi-Modal Detector
Recent multi-modal detectors based on transformers and modality encoders have successfully achieved impressive results on end-to-end visual object detection conditioned on a raw text query. However, they require a large model size and an enormous amount of computations to achieve high performance, which makes it difficult to deploy mobile applications that are limited by tight hardware resources. In this paper, we present a Lightweight modulated detector, Lite-MDETR, to facilitate efficient end-to-end multi-modal understanding on mobile devices. The key primitive is that Dictionary-Lookup-Transformormations (DLT) is proposed to replace Linear Transformation (LT) in multi-modal detectors where each weight in Linear Transformation (LT) is approximately factorized into a smaller dictionary, index, and coefficient. This way, the enormous linear projection with weights is converted into lite linear projection with dictionaries, a few lookups and scalings with indices and coefficients. DLT can be directly applied to pre-trained detectors, removing the need to perform expensive training from scratch. To tackle the challenging training of DLT due to the non-differentiable index, we convert the index and coefficient into a sparse matrix, train this sparse matrix during the fine-tuning phase, and recover it back to index and coefficient during the inference phase. Extensive experiments on several tasks such as phrase grounding, referring expression comprehension and segmentation show that our Lite-MDETR achieves similar detection accuracy to the prior multi-modal detectors with ~ 4.1xmodel size reduction.
Code (0)
등록된 구현이 없습니다.
Tasks
object-detectionObject DetectionPhrase GroundingReferring ExpressionReferring Expression ComprehensionSimilar Papers 제목 키워드 기반
A Lightweight Modular Framework for Low-Cost Open-Vocabulary Object Detection Training
Object detection is a fundamental challenge in computer vision, centered on recognizing objects within images, with diverse applications in areas like image analysis, robotics, and autonomous vehicles. Although existing …
Autonomous VehiclesComputational Efficiencyobject-detectionObject Detection+5MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding
Multi-modal reasoning systems rely on a pre-trained object detector to extract regions of interest from the image. However, this crucial module is typically used as a black box, trained independently of the downstream ta…
Generalized Referring Expression ComprehensionPhrase GroundingQuestion AnsweringReferring Expression+8MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Multi-modal reasoning systems rely on a pre-trained object detector to extract regions of interest from the image. However, this crucial module is typically used as a black box, trained independently of the downstrea…
Phrase GroundingQuestion AnsweringReferring ExpressionReferring Expression Comprehension+4Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding
Multimodal transformer exhibits high capacity and flexibility to align image and text for visual grounding. However, the existing encoder-only grounding framework (e.g., TransVG) suffers from heavy computation due to the…
DecoderVisual GroundingExploring Modulated Detection Transformer as a Tool for Action Recognition in Videos
During recent years transformers architectures have been growing in popularity. Modulated Detection Transformer (MDETR) is an end-to-end multi-modal understanding model that performs tasks such as phase grounding, referr…
Action DetectionAction RecognitionAction Recognition In VideosQuestion Answering+5