Image Captioning: Transforming Objects into Words
Image captioning models typically follow an encoder-decoder architecture which uses abstract image feature vectors as input to the encoder. One of the most successful algorithms uses feature vectors extracted from the region proposals obtained from an object detector. In this work we introduce the Object Relation Transformer, that builds upon this approach by explicitly incorporating information about the spatial relationship between input detected objects through geometric attention. Quantitative and qualitative results demonstrate the importance of such geometric attention for image captioning, leading to improvements on all common captioning metrics on the MS-COCO dataset.
Code (4)
Tasks
DecoderImage CaptioningObjectMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning
Video captioning combines video understanding and language generation. Different from image captioning that describes a static image with details of almost every object, video captioning usually considers a sequence of f…
AttributeCaption GenerationImage CaptioningText Generation+2Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
Image captioning often requires a large set of training image-sentence pairs. In practice, however, acquiring sufficient training pairs is always expensive, making the recent captioning models limited in their ability to…
DecoderImage CaptioningObject RecognitionSentenceRORPCap: Retrieval-based Objects and Relations Prompt for Image Captioning
Image captioning aims to generate natural language descriptions for input images in an open-form manner. To accurately generate descriptions related to the image, a critical step in image captioning is to identify object…
Image CaptioningText RetrievalTop-Down Framework for Weakly-supervised Grounded Image Captioning
Weakly-supervised grounded image captioning (WSGIC) aims to generate the caption and ground (localize) predicted object words in the input image without using bounding box supervision. Recent two-stage solutions mostly a…
Image CaptioningMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONObject+1Guided Open Vocabulary Image Captioning with Constrained Beam Search
Existing image captioning models do not generalize well to out-of-domain images containing novel scenes or objects. This limitation severely hinders the use of these models in real world applications dealing with images …
Image CaptioningTAGWord Embeddings