Explore the Potential Performance of Vision-and-Language Navigation Model: a Snapshot Ensemble Method
Given an instruction in a natural language, the vision-and-language navigation (VLN) task requires a navigation model to match the instruction to its visual surroundings and then move to the correct destination. It has been difficult to build VLN models that can generalize as well as humans. In this paper, we provide a new perspective that accommodates the potential variety of interpretations of verbal instructions. We discovered that snapshots of a VLN model, i.e., model versions based on parameters saved at various intervals during its training, behave significantly differently even when their navigation success rates are almost the same. We thus propose a snapshot-based ensemble solution that leverages predictions provided by multiple snapshots. Our approach is effective and generalizable, and can be applied to ensemble snapshots from different models. Constructed on the mixed snapshots of the existing state-of-the-art (SOTA) RecBERT and HAMT models, our proposed ensemble achieves new SOTA performance in the R2R Dataset Challenge in the single-run setting.
Code (0)
등록된 구현이 없습니다.
Tasks
Vision and Language NavigationSimilar Papers 제목 키워드 기반
VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation
This paper explores the potential of Large Language Models(LLMs) in zero-shot anomaly detection for safe visual navigation. With the assistance of the state-of-the-art real-time open-world object detection model Yolo-Wor…
Anomaly Detectionobject-detectionObject DetectionOpen-vocabulary object detection+5LangNav: Language as a Perceptual Representation for Navigation
We explore the use of language as a perceptual representation for vision-and-language navigation (VLN), with a focus on low-data settings. Our approach uses off-the-shelf vision systems for image captioning and object de…
Image CaptioningLanguage ModelingLanguage ModellingLarge Language Model+3Explore the Potential Performance of Vision-and-Language Navigation Model: a Snapshot Ensemble Method
Vision-and-Language Navigation (VLN) is a challenging task in the field of artificial intelligence. Although massive progress has been made in this task over the past few years attributed to breakthroughs in deep vision …
Vision and Language NavigationSim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environments
Recent work in Vision-and-Language Navigation (VLN) has presented two environmental paradigms with differing realism -- the standard VLN setting built on topological environments where navigation is abstracted away, and …
NavigateVision and Language NavigationImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
Visual navigation is a fundamental capability for autonomous home-assistance robots, enabling long-horizon tasks such as object search. While recent methods have leveraged Large Language Models (LLMs) to incorporate comm…
Spatial ReasoningVisual Navigation