Mx2M: Masked Cross-Modality Modeling in Domain Adaptation for 3D Semantic Segmentation
Existing methods of cross-modal domain adaptation for 3D semantic segmentation predict results only via 2D-3D complementarity that is obtained by cross-modal feature matching. However, as lacking supervision in the target domain, the complementarity is not always reliable. The results are not ideal when the domain gap is large. To solve the problem of lacking supervision, we introduce masked modeling into this task and propose a method Mx2M, which utilizes masked cross-modality modeling to reduce the large domain gap. Our Mx2M contains two components. One is the core solution, cross-modal removal and prediction (xMRP), which makes the Mx2M adapt to various scenarios and provides cross-modal self-supervision. The other is a new way of cross-modal feature matching, the dynamic cross-modal filter (DxMF) that ensures the whole method dynamically uses more suitable 2D-3D complementarity. Evaluation of the Mx2M on three DA scenarios, including Day/Night, USA/Singapore, and A2D2/SemanticKITTI, brings large improvements over previous methods on many metrics.
Code (0)
등록된 구현이 없습니다.
Tasks
3D Semantic SegmentationDomain AdaptationSemantic SegmentationSimilar Papers 제목 키워드 기반
UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modeling
Retinal foundation models aim to learn generalizable representations from diverse retinal images, facilitating label-efficient model adaptation across various ophthalmic tasks. Despite their success, current retinal foun…
Representation LearningTest-Time Adaptation for Visual Document Understanding
For visual document understanding (VDU), self-supervised pretraining has been shown to successfully generate transferable representations, yet, effective adaptation of such representations to distribution shifts at test-…
document understandingDomain AdaptationLanguage ModelingLanguage Modelling+6Masked Vision and Language Modeling for Multi-modal Representation Learning
In this paper, we study how to use masked signal modeling in vision and language (V+L) representation learning. Instead of developing masked language modeling (MLM) and masked image modeling (MIM) independently, we propo…
cross-modal alignmentLanguage ModelingLanguage ModellingMasked Language Modeling+1MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning
We introduce MOSAIC (Masked Objective with Selective Adaptation for In-domain Contrastive learning), a multi-stage framework for domain adaptation of text embedding models that incorporates joint domain-specific masked s…
Contrastive LearningDomain AdaptationMLIM: Vision-and-Language Model Pre-training with Masked Language and Image Modeling
Vision-and-Language Pre-training (VLP) improves model performance for downstream tasks that require image and text inputs. Current VLP approaches differ on (i) model architecture (especially image embedders), (ii) loss f…
Image ReconstructionLanguage ModelingLanguage ModellingMasked Language Modeling