Language-enhanced RNR-Map: Querying Renderable Neural Radiance Field maps with natural language
We present Le-RNR-Map, a Language-enhanced Renderable Neural Radiance map for Visual Navigation with natural language query prompts. The recently proposed RNR-Map employs a grid structure comprising latent codes positioned at each pixel. These latent codes, which are derived from image observation, enable: i) image rendering given a camera pose, since they are converted to Neural Radiance Field; ii) image navigation and localization with astonishing accuracy. On top of this, we enhance RNR-Map with CLIP-based embedding latent codes, allowing natural language search without additional label data. We evaluate the effectiveness of this map in single and multi-object searches. We also investigate its compatibility with a Large Language Model as an "affordance query resolver". Code and videos are available at https://intelligolabs.github.io/Le-RNR-Map/
Code (1)
Tasks
Language ModelingLanguage ModellingLarge Language ModelVisual NavigationSimilar Papers 제목 키워드 기반
RNR-Nav: A Real-World Visual Navigation System Using Renderable Neural Radiance Maps
We propose a novel visual localization and navigation framework for real-world environments directly integrating observed visual information into the bird-eye-view map. While the renderable neural radiance map (RNR-Map) …
Visual LocalizationVisual NavigationRenderable Neural Radiance Map for Visual Navigation
We propose a novel type of map for visual navigation, a renderable neural radiance map (RNR-Map), which is designed to contain the overall visual information of a 3D environment. The RNR-Map has a grid form and consists …
DescriptiveVisual LocalizationVisual NavigationEstimating Neural Reflectance Field from Radiance Field using Tree Structures
We present a new method for estimating the Neural Reflectance Field (NReF) of an object from a set of posed multi-view images under unknown lighting. NReF represents 3D geometry and appearance of objects in a disentangle…
3D geometryNeRFBridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images
Comprehensive 3D scene understanding from sparse, unposed images requires a model to recover renderable geometry, open-vocabulary semantics, and free/occupied 3D space without relying on external camera calibration. Rece…
Novel View SynthesisScene UnderstandingNeRFrac: Neural Radiance Fields through Refractive Surface
Neural Radiance Fields (NeRF) is a popular neural expression for novel view synthesis. By querying spatial points and view directions, a multilayer perceptron (MLP) can be trained to output the volume density and rad…
NeRFNovel View Synthesis