The Wallpaper is Ugly: Indoor Localization using Vision and Language
We study the task of locating a user in a mapped indoor environment using natural language queries and images from the environment. Building on recent pretrained vision-language models, we learn a similarity score between text descriptions and images of locations in the environment. This score allows us to identify locations that best match the language query, estimating the user's location. Our approach is capable of localizing on environments, text, and images that were not seen during training. One model, finetuned CLIP, outperformed humans in our evaluation.
Code (0)
등록된 구현이 없습니다.
Tasks
Indoor LocalizationNatural Language QueriesMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Wallpaper Texture Generation and Style Transfer Based on Multi-label Semantics
Textures contain a wealth of image information and are widely used in various fields such as computer graphics and computer vision. With the development of machine learning, the texture synthesis and generation have been…
AttributeStyle TransferTexture SynthesisVision-Based Localization and LLM-based Navigation for Indoor Environments
Indoor navigation remains a complex challenge due to the absence of reliable GPS signals and the architectural intricacies of large enclosed environments. This study presents an indoor localization and navigation approac…
A Variational Auto-Encoder Enabled Multi-Band Channel Prediction Scheme for Indoor Localization
Indoor localization is getting increasing demands for various cutting-edged technologies, like Virtual/Augmented reality and smart home. Traditional model-based localization suffers from significant computational overhea…
Indoor LocalizationAugLy: Data Augmentations for Robustness
We introduce AugLy, a data augmentation library with a focus on adversarial robustness. AugLy provides a wide array of augmentations for multiple modalities (audio, image, text, & video). These augmentations were inspire…
Adversarial RobustnessData AugmentationBeyond Controlled Environments: 3D Camera Re-Localization in Changing Indoor Scenes
Long-term camera re-localization is an important task with numerous computer vision and robotics applications. Whilst various outdoor benchmarks exist that target lighting, weather and seasonal changes, far less attentio…
Camera Relocalization