Deeply Supervised Multimodal Attentional Translation Embeddings for Visual Relationship Detection
Detecting visual relationships, i.e. <Subject, Predicate, Object> triplets, is a challenging Scene Understanding task approached in the past via linguistic priors or spatial information in a single feature branch. We introduce a new deeply supervised two-branch architecture, the Multimodal Attentional Translation Embeddings, where the visual features of each branch are driven by a multimodal attentional mechanism that exploits spatio-linguistic similarities in a low-dimensional space. We present a variety of experiments comparing against all related approaches in the literature, as well as by re-implementing and fine-tuning several of them. Results on the commonly employed VRD dataset [1] show that the proposed method clearly outperforms all others, while we also justify our claims both quantitatively and qualitatively.
Code (1)
Tasks
Relationship DetectionScene UnderstandingTranslationVisual Relationship DetectionSimilar Papers 제목 키워드 기반
Multimodal Machine Translation with Embedding Prediction
Multimodal machine translation is an attractive application of neural machine translation (NMT). It helps computers to deeply understand visual objects and their relations with natural languages. However, multimodal NMT …
Machine TranslationMultimodal Machine TranslationNMTPrediction+3Unified Attentional Generative Adversarial Network for Brain Tumor Segmentation From Multimodal Unpaired Images
In medical applications, the same anatomical structures may be observed in multiple modalities despite the different image characteristics. Currently, most deep models for multimodal segmentation rely on paired registere…
Brain Tumor SegmentationGenerative Adversarial NetworkSegmentationTranslation+1Does Multimodality Help Human and Machine for Translation and Image Captioning?
This paper presents the systems developed by LIUM and CVC for the WMT16 Multimodal Machine Translation challenge. We explored various comparative methods, namely phrase-based systems and attentional recurrent neural netw…
Image CaptioningImage DescriptionMachine TranslationMultimodal Machine Translation+1Mancs: A Multi-task Attentional Network with Curriculum Sampling for Person Re-identification
We propose a novel deep network called Mancs that solves the person re-identification problem from the following aspects: fully utilizing the attention mechanism for the person misalignment problem and properly sampling …
Person Re-IdentificationTowards Neural Machine Translation with Latent Tree Attention
Building models that take advantage of the hierarchical structure of language without a priori annotation is a longstanding goal in natural language processing. We introduce such a model for the task of machine translati…
DecoderMachine Translationreinforcement-learningReinforcement Learning+3