Pix2seq: A Language Modeling Framework for Object Detection
We present Pix2Seq, a simple and generic framework for object detection. Unlike existing approaches that explicitly integrate prior knowledge about the task, we cast object detection as a language modeling task conditioned on the observed pixel inputs. Object descriptions (e.g., bounding boxes and class labels) are expressed as sequences of discrete tokens, and we train a neural network to perceive the image and generate the desired sequence. Our approach is based mainly on the intuition that if a neural network knows about where and what the objects are, we just need to teach it how to read them out. Beyond the use of task-specific data augmentations, our approach makes minimal assumptions about the task, yet it achieves competitive results on the challenging COCO dataset, compared to highly specialized and well optimized detection algorithms.
Code (6)
Tasks
Language ModelingLanguage ModellingObjectobject-detectionObject DetectionSimilar Papers 제목 키워드 기반
Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection
In this paper, we for the first time explore helpful multi-modal contextual knowledge to understand novel categories for open-vocabulary object detection (OVD). The multi-modal contextual knowledge stands for the joint r…
Knowledge DistillationLanguage ModelingLanguage ModellingMasked Language Modeling+5Enhancing Open-Vocabulary Object Detection through Multi-Level Fine-Grained Visual-Language Alignment
Traditional object detection systems are typically constrained to predefined categories, limiting their applicability in dynamic environments. In contrast, open-vocabulary object detection (OVD) enables the identificatio…
Object DetectionLogGPT: Log Anomaly Detection via GPT
Detecting system anomalies based on log data is important for ensuring the security and reliability of computer systems. Recently, deep learning models have been widely used for log anomaly detection. The core idea is to…
Anomaly DetectionLanguage ModelingLanguage ModellingSemi-supervised Multitask Learning for Sequence Labeling
We propose a sequence labeling framework with a secondary training objective, learning to predict surrounding words for every word in the dataset. This language modeling objective incentivises the system to learn general…
ChunkingGrammatical Error DetectionLanguage ModelingLanguage Modelling+5Contextual Object Detection with Multimodal Large Language Models
Recent Multimodal Large Language Models (MLLMs) are remarkable in vision-language tasks, such as image captioning and question answering, but lack the essential perception ability, i.e., object detection. In this work, w…
Cloze TestDecoderImage CaptioningImage Segmentation+5