ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments
For embodied agents, navigation is an important ability but not an isolated goal. Agents are also expected to perform specific tasks after reaching the target location, such as picking up objects and assembling them into a particular arrangement. We combine Vision-and-Language Navigation, assembling of collected objects, and object referring expression comprehension, to create a novel joint navigation-and-assembly task, named ArraMon. During this task, the agent (similar to a PokeMON GO player) is asked to find and collect different target objects one-by-one by navigating based on natural language instructions in a complex, realistic outdoor environment, but then also ARRAnge the collected objects part-by-part in an egocentric grid-layout environment. To support this task, we implement a 3D dynamic environment simulator and collect a dataset (in English; and also extended to Hindi) with human-written navigation and assembling instructions, and the corresponding ground truth trajectories. We also filter the collected instructions via a verification stage, leading to a total of 7.7K task instances (30.8K instructions and paths). We present results for several baseline models (integrated and biased) and metrics (nDTW, CTC, rPOD, and PTC), and the large model-human performance gap demonstrates that our task is challenging and presents a wide scope for future work. Our dataset, simulator, and code are publicly available at: https://arramonunc.github.io
Code (0)
등록된 구현이 없습니다.
Tasks
Referring ExpressionReferring Expression ComprehensionVision and Language NavigationSimilar Papers 제목 키워드 기반
AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects
Assembling objects from parts requires understanding multimodal instructions, linking them to 3D components, and predicting physically plausible 6-DoF motions for each assembly step. Existing datasets focus on simplified…
Pose EstimationOpenMap: Instruction Grounding via Open-Vocabulary Visual-Language Mapping
Grounding natural language instructions to visual observations is fundamental for embodied agents operating in open-world environments. Recent advances in visual-language mapping have enabled generalizable semantic repre…
Green for Go, Red for No: Visual Grounding via Semantic Segmentation for VLA Navigation Policies
Vision-language-action (VLA) models enable robot navigation from natural language and visual goals, but remain susceptible to perceptual distractions and ambiguous scene interpretations. This paper presents the first emp…
Semantic SegmentationRobot NavigationVisual GroundingSource-Target Inference Models for Spatial Instruction Understanding
Models that can execute natural language instructions for situated robotic tasks such as assembly and navigation have several useful applications in homes, offices, and remote scenarios. We study the semantics of spatial…
PositionPosition regressionregressionRepresentation LearningObservation-Graph Interaction and Key-Detail Guidance for Vision and Language Navigation
Vision and Language Navigation (VLN) requires an agent to navigate through environments following natural language instructions. However, existing methods often struggle with effectively integrating visual observations a…
cross-modal alignmentNavigateVision and Language Navigation