F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models
We present F-VLM, a simple open-vocabulary object detection method built upon Frozen Vision and Language Models. F-VLM simplifies the current multi-stage training pipeline by eliminating the need for knowledge distillation or detection-tailored pretraining. Surprisingly, we observe that a frozen VLM: 1) retains the locality-sensitive features necessary for detection, and 2) is a strong region classifier. We finetune only the detector head and combine the detector and VLM outputs for each region at inference time. F-VLM shows compelling scaling behavior and achieves +6.5 mask AP improvement over the previous state of the art on novel categories of LVIS open-vocabulary detection benchmark. In addition, we demonstrate very competitive results on COCO open-vocabulary detection benchmark and cross-dataset transfer detection, in addition to significant training speed-up and compute savings. Code will be released at the https://sites.google.com/view/f-vlm/home
Code (1)
Tasks
Knowledge Distillationobject-detectionObject DetectionOpen-vocabulary object detectionOpen Vocabulary Object DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Multi-modal Queried Object Detection in the Wild
We introduce MQ-Det, an efficient architecture and pre-training strategy design to utilize both textual description with open-set generalization and visual exemplars with rich description granularity as category queries,…
Few-Shot Object DetectionObjectobject-detectionObject Detection+1UniVCD: A New Method for Unsupervised Change Detection in the Open-Vocabulary Era
Change detection (CD) identifies scene changes from multi-temporal observations and is widely used in urban development and environmental monitoring. Most existing CD methods rely on supervised learning, making performan…
Change DetectionContrastive Feature Masking Open-Vocabulary Vision Transformer
We present Contrastive Feature Masking Vision Transformer (CFM-ViT) - an image-text pretraining methodology that achieves simultaneous learning of image- and region-level representation for open-vocabulary object detecti…
Contrastive LearningImage-text Retrievalobject-detectionObject Detection+4Modulating CNN Features with Pre-Trained ViT Representations for Open-Vocabulary Object Detection
Owing to large-scale image-text contrastive training, pre-trained vision language model (VLM) like CLIP shows superior open-vocabulary recognition ability. Most existing open-vocabulary object detectors attempt to utiliz…
object-detectionObject DetectionOpen-vocabulary object detectionOpen Vocabulary Object DetectionInstance Brownian Bridge as Texts for Open-vocabulary Video Instance Segmentation
Temporally locating objects with arbitrary class texts is the primary pursuit of open-vocabulary Video Instance Segmentation (VIS). Because of the insufficient vocabulary of video data, previous methods leverage image-te…
Instance SegmentationSemantic SegmentationVideo Instance Segmentation