Prompt-Guided Transformers for End-to-End Open-Vocabulary Object Detection

2023-03-25 · Hwanjun Song, Jihwan Bang

Prompt-OVD is an efficient and effective framework for open-vocabulary object detection that utilizes class embeddings from CLIP as prompts, guiding the Transformer decoder to detect objects in both base and novel classes. Additionally, our novel RoI-based masked attention and RoI pruning techniques help leverage the zero-shot classification ability of the Vision Transformer-based CLIP, resulting in improved detection performance at minimal computational cost. Our experiments on the OV-COCO and OVLVIS datasets demonstrate that Prompt-OVD achieves an impressive 21.2 times faster inference speed than the first end-to-end open-vocabulary detection method (OV-DETR), while also achieving higher APs than four two-stage-based methods operating within similar inference time ranges. Code will be made available soon.

📄 PDF Abstract BibTeX arXiv:2303.14386

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder object-detection Object Detection Open-vocabulary object detection Open Vocabulary Object Detectionzero-shot-classificationZero-Shot Learning

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Pruning 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Residual Connection 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Position-Wise Feed-Forward Layer 설명 없음

LLM-Guided Agentic Object Detection for Open-World Understanding

2025-07-14 · Furkan Mumcu, Michael J. Jones, Anoop Cherian, Yasin Yilmaz arxiv

Object detection traditionally relies on fixed category sets, requiring costly re-training to handle novel objects. While Open-World and Open-Vocabulary Object Detection (OWOD and OVOD) improve flexibility, OWOD lacks se…

Object Detection

End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Prompting

2024-09-19 · Yongqi Wang, Shuo Yang, Xinxiao wu, Jiebo Luo

Open-vocabulary video visual relationship detection aims to expand video visual relationship detection beyond annotated categories by detecting unseen relationships between both seen and unseen objects in videos. Existin…

Decoder Object object-detection Object Detection+5

VL-SAM-v3: Memory-Guided Visual Priors for Open-World Object Detection

2026-05-05 · Chih-Chung Liu, Zhiwei Lin, Yongtao Wang arxiv

Open-world object detection aims to localize and recognize objects beyond a fixed closed-set label space. It is commonly divided into two categories, i.e., open-vocabulary detection, which assumes a predefined category l…

Object Detection

YOLOE-26: Integrating YOLO26 with YOLOE for Real-Time Open-Vocabulary Instance Segmentation

2026-01-29 · Ranjan Sapkota, Manoj Karkee arxiv

This paper presents YOLOE-26, a unified framework that integrates the deployment-optimized YOLO26(or YOLOv26) architecture with the open-vocabulary learning paradigm of YOLOE for real-time open-vocabulary instance segmen…

Instance Segmentation

Prompt-Guided Mask Proposal for Two-Stage Open-Vocabulary Segmentation

2024-12-13 · Yu-Jhe Li, Xinyang Zhang, Kun Wan, Lantao Yu 외

We tackle the challenge of open-vocabulary segmentation, where we need to identify objects from a wide range of categories in different environments, using text prompts as our input. To overcome this challenge, existing …