simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions
The encode-decoder framework has shown recent success in image captioning. Visual attention, which is good at detailedness, and semantic attention, which is good at comprehensiveness, have been separately proposed to ground the caption on the image. In this paper, we propose the Stepwise Image-Topic Merging Network (simNet) that makes use of the two kinds of attention at the same time. At each time step when generating the caption, the decoder adaptively merges the attentive information in the extracted topics and the image according to the generated context, so that the visual information and the semantic information can be effectively combined. The proposed approach is evaluated on two benchmark datasets and reaches the state-of-the-art performances.(The code is available at https://github.com/lancopku/simNet)
Code (1)
Tasks
DecoderImage CaptioningSimilar Papers 제목 키워드 기반
SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo
Robot manipulation of unknown objects in unstructured environments is a challenging problem due to the variety of shapes, materials, arrangements and lighting conditions. Even with large-scale real-world data collection,…
Keypoint DetectionObjectobject-detectionObject Detection+3NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
We present SimNet, an AI-driven multi-physics simulation framework, to accelerate simulations across a wide range of disciplines in science and engineering. Compared to traditional numerical solvers, SimNet addresses a w…
GPUDeep SimNets
We present a deep layered architecture that generalizes convolutional neural networks (ConvNets). The architecture, called SimNets, is driven by two operators: (i) a similarity function that generalizes inner-product, an…
BOLDSimNet: Examining Brain Network Similarity between Task and Resting-State fMRI
Traditional causal connectivity methods in task-based and resting-state functional magnetic resonance imaging (fMRI) face challenges in accurately capturing directed information flow due to their sensitivity to noise and…
SimNets: A Generalization of Convolutional Networks
We present a deep layered architecture that generalizes classical convolutional neural networks (ConvNets). The architecture, called SimNets, is driven by two operators, one being a similarity function whose family conta…