CPTR: Full Transformer Network for Image Captioning
In this paper, we consider the image captioning task from a new sequence-to-sequence prediction perspective and propose CaPtion TransformeR (CPTR) which takes the sequentialized raw images as the input to Transformer. Compared to the "CNN+Transformer" design paradigm, our model can model global context at every encoder layer from the beginning and is totally convolution-free. Extensive experiments demonstrate the effectiveness of the proposed model and we surpass the conventional "CNN+Transformer" methods on the MSCOCO dataset. Besides, we provide detailed visualizations of the self-attention between patches in the encoder and the "words-to-patches" attention in the decoder thanks to the full Transformer architecture.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderImage CaptioningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ReFormer: The Relational Transformer for Image Captioning
Image captioning is shown to be able to achieve a better performance by using scene graphs to represent the relations of objects in the image. The current captioning encoders generally use a Graph Convolutional Net (GCN)…
Graph GenerationImage CaptioningRelationScene Graph Generation+1Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning
Existing image captioning methods just focus on understanding the relationship between objects or instances in a single image, without exploring the contextual correlation existed among contextual image. In this paper, w…
Image CaptioningObjectMeshed-Memory Transformer for Image Captioning
Transformer-based architectures represent the state of the art in sequence modeling tasks like machine translation and language understanding. Their applicability to multi-modal contexts like image captioning, however, i…
Image CaptioningMachine TranslationText GenerationTranslationImage Captioning as Neural Machine Translation Task in SOCKEYE
Image captioning is an interdisciplinary research problem that stands between computer vision and natural language processing. The task is to generate a textual description of the content of an image. The typical model u…
DecoderImage CaptioningMachine TranslationSentence+1Transformer based Multitask Learning for Image Captioning and Object Detection
In several real-world scenarios like autonomous navigation and mobility, to obtain a better visual understanding of the surroundings, image captioning and object detection play a crucial role. This work introduces a nove…
Autonomous NavigationImage CaptioningObjectobject-detection+1