DAP: Domain-aware Prompt Learning for Vision-and-Language Navigation
Following language instructions to navigate in unseen environments is a challenging task for autonomous embodied agents. With strong representation capabilities, pretrained vision-and-language models are widely used in VLN. However, most of them are trained on web-crawled general-purpose datasets, which incurs a considerable domain gap when used for VLN tasks. To address the problem, we propose a novel and model-agnostic domain-aware prompt learning (DAP) framework. For equipping the pretrained models with specific object-level and scene-level cross-modal alignment in VLN tasks, DAP applies a low-cost prompt tuning paradigm to learn soft visual prompts for extracting in-domain image semantics. Specifically, we first generate a set of in-domain image-text pairs with the help of the CLIP model. Then we introduce soft visual prompts in the input space of the visual encoder in a pretrained model. DAP injects in-domain visual knowledge into the visual encoder of the pretrained model in an efficient way. Experimental results on both R2R and REVERIE show the superiority of DAP compared to existing state-of-the-art methods.
Code (0)
등록된 구현이 없습니다.
Tasks
cross-modal alignmentNavigatePrompt LearningVision and Language NavigationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Prompt-based Context- and Domain-aware Pretraining for Vision and Language Navigation
Pretrained visual-language models have extensive world knowledge and are widely used in visual and language navigation (VLN). However, they are not sensitive to indoor scenarios for VLN tasks. Another challenge for VLN i…
Contrastive Learningcross-modal alignmentVision and Language NavigationWorld KnowledgeHistory-Augmented Vision-Language Models for Frontier-Based Zero-Shot Object Navigation
Object Goal Navigation (ObjectNav) challenges robots to find objects in unseen environments, demanding sophisticated reasoning. While Vision-Language Models (VLMs) show potential, current ObjectNav methods often employ t…
Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration
Vision-language navigation (VLN) is a challenging task due to its large searching space in the environment. To address this problem, previous works have proposed some methods of fine-tuning a large model that pretrained …
Domain AdaptationVision-Language NavigationVisual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration
Vision-language navigation (VLN) is a challenging task due to its large searching space in the environment. To address this problem, previous works have proposed some methods of fine-tuning a large model that pretrained …
Domain AdaptationVision-Language NavigationWhy Only Text: Empowering Vision-and-Language Navigation with Multi-modal Prompts
Current Vision-and-Language Navigation (VLN) tasks mainly employ textual instructions to guide agents. However, being inherently abstract, the same textual instruction can be associated with different visual signals, cau…
NavigateVision and Language Navigation