World knowledge-enhanced Reasoning Using Instruction-guided Interactor in Autonomous Driving
The Multi-modal Large Language Models (MLLMs) with extensive world knowledge have revitalized autonomous driving, particularly in reasoning tasks within perceivable regions. However, when faced with perception-limited areas (dynamic or static occlusion regions), MLLMs struggle to effectively integrate perception ability with world knowledge for reasoning. These perception-limited regions can conceal crucial safety information, especially for vulnerable road users. In this paper, we propose a framework, which aims to improve autonomous driving performance under perceptionlimited conditions by enhancing the integration of perception capabilities and world knowledge. Specifically, we propose a plug-and-play instruction-guided interaction module that bridges modality gaps and significantly reduces the input sequence length, allowing it to adapt effectively to multi-view video inputs. Furthermore, to better integrate world knowledge with driving-related tasks, we have collected and refined a large-scale multi-modal dataset that includes 2 million natural language QA pairs, 1.7 million grounding task data. To evaluate the model's utilization of world knowledge, we introduce an object-level risk assessment dataset comprising 200K QA pairs, where the questions necessitate multi-step reasoning leveraging world knowledge for resolution. Extensive experiments validate the effectiveness of our proposed method.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingWorld KnowledgeSimilar Papers 제목 키워드 기반
KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigation
Vision-and-language navigation (VLN) is the task to enable an embodied agent to navigate to a remote location following the natural language instruction in real scenes. Most of the previous approaches utilize the entire …
NavigateVision and Language NavigationKnowledge-enhanced Iterative Instruction Generation and Reasoning for Knowledge Base Question Answering
Multi-hop Knowledge Base Question Answering(KBQA) aims to find the answer entity in a knowledge base which is several hops from the topic entity mentioned in the question. Existing Retrieval-based approaches first genera…
Knowledge Base Question AnsweringQuestion AnsweringRetrievalWorldEdit: Towards Open-World Image Editing with a Knowledge-Informed Benchmark
Recent advances in image editing models have demonstrated remarkable capabilities in executing explicit instructions, such as attribute manipulation, style transfer, and pose synthesis. However, these models often face c…
Instruction FollowingStyle TransferImage EditingSurfer: Progressive Reasoning with World Models for Robotic Manipulation
Considering how to make the model accurately understand and follow natural language instructions and perform actions consistent with world knowledge is a key challenge in robot manipulation. This mainly includes human fu…
Decision MakingMuJoCoNatural Language UnderstandingRobot Manipulation+1MobileVLA-R1 2.0: RL-Enhanced Reasoning for Mobile Robot Control
Grounding natural-language instructions into reliable and executable actions remains a fundamental challenge for vision-language-action (VLA) systems on mobile robots, due to the persistent gap between high-level semanti…
Reinforcement LearningInstruction FollowingMultimodal ReasoningDecision Making