Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds
Dense captioning in 3D point clouds is an emerging vision-and-language task involving object-level 3D scene understanding. Apart from coarse semantic class prediction and bounding box regression as in traditional 3D object detection, 3D dense captioning aims at producing a further and finer instance-level label of natural language description on visual appearance and spatial relations for each scene object of interest. To detect and describe objects in a scene, following the spirit of neural machine translation, we propose a transformer-based encoder-decoder architecture, namely SpaCap3D, to transform objects into descriptions, where we especially investigate the relative spatiality of objects in 3D scenes and design a spatiality-guided encoder via a token-to-token spatial relation learning objective and an object-centric decoder for precise and spatiality-enhanced object caption generation. Evaluated on two benchmark datasets, ScanRefer and ReferIt3D, our proposed SpaCap3D outperforms the baseline method Scan2Cap by 4.94% and 9.61% in CIDEr@0.5IoU, respectively. Our project page with source code and supplementary files is available at https://SpaCap3D.github.io/ .
Code (1)
Tasks
3D dense captioning3D Object DetectionCaption GenerationDecoderDense CaptioningMachine TranslationObjectobject-detectionObject DetectionScene UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning
3D dense captioning aims to describe individual objects by natural language in 3D scenes, where 3D scenes are usually represented as RGB-D scans or point clouds. However, only exploiting single modal information, e.g., p…
3D dense captioningDense CaptioningKnowledge DistillationTransfer LearningEnd-to-End Dense Video Captioning with Masked Transformer
Dense video captioning aims to generate text descriptions for all events in an untrimmed video. This involves both detecting and describing events. Therefore, all previous methods on dense video captioning tackle this pr…
DecoderDense Video CaptioningVideo CaptioningUniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding
Performing 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships. However, despite some previous attempts on connecting these two related tasks wit…
3D dense captioning3D visual groundingDense CaptioningVisual GroundingA Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer
Dense video captioning aims to localize and describe important events in untrimmed videos. Existing methods mainly tackle this task by exploiting only visual features, while completely neglecting the audio track. Only a …
Dense Video CaptioningTemporal Action Proposal GenerationVideo CaptioningSee It All: Contextualized Late Aggregation for 3D Dense Captioning
3D dense captioning is a task to localize objects in a 3D scene and generate descriptive sentences for each object. Recent approaches in 3D dense captioning have adopted transformer encoder-decoder frameworks from object…
3D dense captioningAllAttributeCaption Generation+6