GRiT: A Generative Region-to-text Transformer for Object Understanding
This paper presents a Generative RegIon-to-Text transformer, GRiT, for object understanding. The spirit of GRiT is to formulate object understanding as <region, text> pairs, where region locates objects and text describes objects. For example, the text in object detection denotes class names while that in dense captioning refers to descriptive sentences. Specifically, GRiT consists of a visual encoder to extract image features, a foreground object extractor to localize objects, and a text decoder to generate open-set object descriptions. With the same model architecture, GRiT can understand objects via not only simple nouns, but also rich descriptive sentences including object attributes or actions. Experimentally, we apply GRiT to object detection and dense captioning tasks. GRiT achieves 60.4 AP on COCO 2017 test-dev for object detection and 15.5 mAP on Visual Genome for dense captioning. Code is available at https://github.com/JialianW/GRiT
Code (1)
Tasks
DecoderDense CaptioningDescriptiveObjectobject-detectionObject DetectionSimilar Papers 제목 키워드 기반
GRIT: Faster and Better Image captioning Transformer Using Dual Visual Features
Current state-of-the-art methods for image captioning employ region-based features, as they provide object-level information that is essential to describe the content of images; they are usually extracted by an object de…
Image CaptioningMachine Intelligence that Understands Visual and Linguistic Information and Interacts with Humans and Environments
Advancements at the intersection of computer vision and natural language processing are crucial for applications like assistive tech, multimedia querying, and robotics. This dissertation proposes novel architectures to i…
Instruction FollowingImage CaptioningVisual DialogImage captioning for Brazilian Portuguese using GRIT model
This work presents the early development of a model of image captioning for the Brazilian Portuguese language. We used the GRIT (Grid - and Region-based Image captioning Transformer) model to accomplish this work. GRIT i…
Image CaptioningmodelGRIT: Generative Role-filler Transformers for Document-level Event Entity Extraction
We revisit the classic problem of document-level role-filler entity extraction (REE) for template filling. We argue that sentence-level approaches are ill-suited to the task and introduce a generative transformer-based e…
DecoderRole-filler Entity ExtractionSentenceENCORE: Entropy-Guided Cropping and Attention Regularization for Robust Vision--Language Understanding
Vision-Language Models (VLMs) perform well on diverse vision-language tasks, but transformer-based visual encoders split images into fixed-resolution sub-images, compromising object integrity in lightweight VLMs. Existin…