CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
We propose CLIP-Fields, an implicit scene model that can be used for a variety of tasks, such as segmentation, instance identification, semantic search over space, and view localization. CLIP-Fields learns a mapping from spatial locations to semantic embedding vectors. Importantly, we show that this mapping can be trained with supervision coming only from web-image and web-text trained models such as CLIP, Detic, and Sentence-BERT; and thus uses no direct human supervision. When compared to baselines like Mask-RCNN, our method outperforms on few-shot instance identification or semantic segmentation on the HM3D dataset with only a fraction of the examples. Finally, we show that using CLIP-Fields as a scene memory, robots can perform semantic navigation in real-world environments. Our code and demonstration videos are available here: https://mahis.life/clip-fields
Code (2)
Tasks
SegmentationSemantic SegmentationSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Set-CLIP: Exploring Aligned Semantic From Low-Alignment Multimodal Data Through A Distribution View
Multimodal fusion breaks through the boundaries between diverse modalities and has already achieved notable performances. However, in many specialized fields, it is struggling to obtain sufficient alignment data for trai…
A weakly supervised sequence tagging and grammar induction approach to semantic frame slot filling
This paper describes continuing work on semantic frame slot filling for a command and control task using a weakly-supervised approach. We investigate the advantages of using retraining techniques that take the output of …
slot-fillingSlot FillingFrozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation
Weakly supervised semantic segmentation has witnessed great achievements with image-level labels. Several recent approaches use the CLIP model to generate pseudo labels for training an individual segmentation model, whil…
DecoderSegmentationSemantic SegmentationWeakly supervised Semantic Segmentation+1Weakly-supervised Learning for Physics-informed Neural Motion Planning via Sparse Roadmap
The motion planning problem requires finding a collision-free path between start and goal configurations in high-dimensional, cluttered spaces. Recent learning-based methods offer promising solutions, with self-supervise…
Motion PlanningVL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations
We present Visual-Language Fields (VL-Fields), a neural implicit spatial representation that enables open-vocabulary semantic queries. Our model encodes and fuses the geometry of a scene with vision-language trained late…
SegmentationSemantic Segmentation