Scan2Cap: Context-aware Dense Captioning in RGB-D Scans
We introduce the task of dense captioning in 3D scans from commodity RGB-D sensors. As input, we assume a point cloud of a 3D scene; the expected output is the bounding boxes along with the descriptions for the underlying objects. To address the 3D object detection and description problems, we propose Scan2Cap, an end-to-end trained method, to detect objects in the input scene and describe them in natural language. We use an attention mechanism that generates descriptive tokens while referring to the related components in the local context. To reflect object relations (i.e. relative spatial relations) in the generated captions, we use a message passing graph module to facilitate learning object relation features. Our method can effectively localize and describe 3D objects in scenes from the ScanRefer dataset, outperforming 2D baseline methods by a significant margin (27.61% CiDEr@0.5IoUimprovement).
Code (0)
등록된 구현이 없습니다.
Tasks
3D dense captioning3D Object Detection3D Question Answering (3D-QA)Dense CaptioningDescriptiveObjectobject-detectionObject DetectionSimilar Papers 제목 키워드 기반
3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
3D vision-language grounding (3D-VL) is an emerging field that aims to connect the 3D physical world with natural language, which is crucial for achieving embodied intelligence. Current 3D-VL models rely heavily on sophi…
3D Question Answering (3D-QA)Dense CaptioningQuestion AnsweringText Matching+2Gated-Dilated Networks for Lung Nodule Classification in CT scans
Different types of Convolutional Neural Networks (CNNs) have been applied to detect cancerous lung nodules from computed tomography (CT) scans. However, the size of a nodule is very diverse and can range anywhere between…
ClassificationComputed Tomography (CT)General ClassificationLung Nodule ClassificationEnd-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scans
We present a novel, end-to-end approach to align CAD models to an 3D scan of a scene, enabling transformation of a noisy, incomplete 3D scan to a compact, CAD reconstruction with clean, complete object geometry. Our main…
CAD ReconstructionObjectRetrievalX-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning
3D dense captioning aims to describe individual objects by natural language in 3D scenes, where 3D scenes are usually represented as RGB-D scans or point clouds. However, only exploiting single modal information, e.g., p…
3D dense captioningDense CaptioningKnowledge DistillationTransfer LearningHierarchical Context-aware Network for Dense Video Event Captioning
Dense video event captioning aims to generate a sequence of descriptive captions for each event in a long untrimmed video. Video-level context provides important information and facilities the model to generate consisten…
Descriptive