D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding
Recent studies on dense captioning and visual grounding in 3D have achieved impressive results. Despite developments in both areas, the limited amount of available 3D vision-language data causes overfitting issues for 3D visual grounding and 3D dense captioning methods. Also, how to discriminatively describe objects in complex 3D environments is not fully studied yet. To address these challenges, we present D3Net, an end-to-end neural speaker-listener architecture that can detect, describe and discriminate. Our D3Net unifies dense captioning and visual grounding in 3D in a self-critical manner. This self-critical property of D3Net also introduces discriminability during object caption generation and enables semi-supervised training on ScanNet data with partially annotated descriptions. Our method outperforms SOTA methods in both tasks on the ScanRefer dataset, surpassing the SOTA 3D dense captioning method by a significant margin.
Code (0)
등록된 구현이 없습니다.
Tasks
3D dense captioning3D visual groundingCaption GenerationDense CaptioningVisual GroundingSimilar Papers 제목 키워드 기반
Pragmatic Inference with a CLIP Listener for Contrastive Captioning
We propose a simple yet effective and robust method for contrastive captioning: generating discriminative captions that distinguish target images from very similar alternative distractor images. Our approach is built on …
Structural and Functional Decomposition for Personality Image Captioning in a Communication Game
Personality image captioning (PIC) aims to describe an image with a natural language caption given a personality trait. In this work, we introduce a novel formulation for PIC based on a communication game between a speak…
Caption GenerationImage CaptioningLanguage ModelingLanguage ModellingA Joint Speaker-Listener-Reinforcer Model for Referring Expressions
Referring expressions are natural language constructions used to identify particular objects within a scene. In this paper, we propose a unified framework for the tasks of referring expression comprehension and generatio…
Referring ExpressionReferring Expression ComprehensionEmergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Data
Image captioning is an important problem in developing various AI systems, and these tasks require large volumes of annotated images to train the models. Since all existing labelled datasets are already used for training…
Multi-agent Reinforcement LearningImage CaptioningUnified Pragmatic Models for Generating and Following Instructions
We show that explicit pragmatic inference aids in correctly generating and following natural language instructions for complex, sequential tasks. Our pragmatics-enabled models reason about why speakers produce certain in…
Text Generation