PointCloud-Text Matching: Benchmark Datasets and a Baseline
In this paper, we present and study a new instance-level retrieval task: PointCloud-Text Matching~(PTM), which aims to find the exact cross-modal instance that matches a given point-cloud query or text query. PTM could be applied to various scenarios, such as indoor/urban-canyon localization and scene retrieval. However, there exists no suitable and targeted dataset for PTM in practice. Therefore, we construct three new PTM benchmark datasets, namely 3D2T-SR, 3D2T-NR, and 3D2T-QA. We observe that the data is challenging and with noisy correspondence due to the sparsity, noise, or disorder of point clouds and the ambiguity, vagueness, or incompleteness of texts, which make existing cross-modal matching methods ineffective for PTM. To tackle these challenges, we propose a PTM baseline, named Robust PointCloud-Text Matching method (RoMa). RoMa consists of two modules: a Dual Attention Perception module (DAP) and a Robust Negative Contrastive Learning module (RNCL). Specifically, DAP leverages token-level and feature-level attention to adaptively focus on useful local and global features, and aggregate them into common representations, thereby reducing the adverse impact of noise and ambiguity. To handle noisy correspondence, RNCL divides negative pairs, which are much less error-prone than positive pairs, into clean and noisy subsets, and assigns them forward and reverse optimization directions respectively, thus enhancing robustness against noisy correspondence. We conduct extensive experiments on our benchmarks and demonstrate the superiority of our RoMa.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningRetrievalText MatchingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DeSPITE: Exploring Contrastive Deep Skeleton-Pointcloud-IMU-Text Embeddings for Advanced Point Cloud Human Activity Understanding
Despite LiDAR (Light Detection and Ranging) being an effective privacy-preserving alternative to RGB cameras to perceive human activities, it remains largely underexplored in the context of multi-modal contrastive pre-tr…
Activity RecognitionHuman Activity RecognitionMoment RetrievalPerson Re-Identification+2Multi-Resolution Graph Neural Network for Large-Scale Pointcloud Segmentation
In this paper, we propose a multi-resolution deep-learning architecture to semantically segment dense large-scale pointclouds. Dense pointcloud data require a computationally expensive feature encoding process before sem…
GPUGraph Neural NetworkSegmentationSemantic SegmentationMuGNet: Multi-Resolution Graph Neural Network for Large-Scale Pointcloud Segmentation
In this paper, we propose a multi-resolution deep-learning architecture to semantically segment dense large-scale pointclouds. Dense pointcloud data require a computationally expensive feature encoding process before sem…
GPUGraph Neural NetworkSegmentationSemantic SegmentationModeling Urban Scenes From Pointclouds
We present a method for Modeling Urban Scenes from Pointclouds (MUSP). In contrast to existing approaches, MUSP is robust, scalable and provides a more complete description by not making a Manhattan-World assumption and …
Scene SegmentationLearning task-specific features for 3D pointcloud graph creation
Processing 3D pointclouds with Deep Learning methods is not an easy task. A common choice is to do so with Graph Neural Networks, but this framework involves the creation of edges between points, which are explicitly not…