Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planning
Long-horizon task planning is essential for the development of intelligent assistive and service robots. In this work, we investigate the applicability of a smaller class of large language models (LLMs), specifically GPT-2, in robotic task planning by learning to decompose tasks into subgoal specifications for a planner to execute sequentially. Our method grounds the input of the LLM on the domain that is represented as a scene graph, enabling it to translate human requests into executable robot plans, thereby learning to reason over long-horizon tasks, as encountered in the ALFRED benchmark. We compare our approach with classical planning and baseline methods to examine the applicability and generalizability of LLM-based planners. Our findings suggest that the knowledge stored in an LLM can be effectively grounded to perform long-horizon task planning, demonstrating the promising potential for the future application of neuro-symbolic planning methods in robotics.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingTask PlanningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Graphhopper: Multi-Hop Scene Graph Reasoning for Visual Question Answering
Visual Question Answering (VQA) is concerned with answering free-form questions about an image. Since it requires a deep semantic and linguistic understanding of the question and the ability to associate it with various …
NavigateQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Understanding the Role of Scene Graphs in Visual Question Answering
Visual Question Answering (VQA) is of tremendous interest to the research community with important applications such as aiding visually impaired users and image-based search. In this work, we explore the use of scene gra…
Graph GenerationQuestion AnsweringScene Graph GenerationVisual Question Answering+1SelfGraphVQA: A Self-Supervised Graph Neural Network for Scene-based Question Answering
The intersection of vision and language is of major interest due to the increased focus on seamless integration between recognition and reasoning. Scene graphs (SGs) have emerged as a useful tool for multimodal image ana…
Graph Neural NetworkQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)Visual Graphs from Motion (VGfM): Scene understanding with object geometry reasoning
Recent approaches on visual scene understanding attempt to build a scene graph -- a computational representation of objects and their pairwise relationships. Such rich semantic representation is very appealing, yet diffi…
3d scene graph generationGraph GenerationScene Graph GenerationScene UnderstandingVariational Amodal Object Completion
In images of complex scenes, objects are often occluding each other which makes perception tasks such as object detection and tracking, or robotic control tasks such as planning, challenging. To facilitate downstream tas…
Objectobject-detectionObject Detection