VirtualHome: Simulating Household Activities via Programs
In this paper, we are interested in modeling complex activities that occur in a typical household. We propose to use programs, i.e., sequences of atomic actions and interactions, as a high level representation of complex tasks. Programs are interesting because they provide a non-ambiguous representation of a task, and allow agents to execute them. However, nowadays, there is no database providing this type of information. Towards this goal, we first crowd-source programs for a variety of activities that happen in people's homes, via a game-like interface used for teaching kids how to code. Using the collected dataset, we show how we can learn to extract programs directly from natural language descriptions or from videos. We then implement the most common atomic (inter)actions in the Unity3D game engine, and use our programs to "drive" an artificial agent to execute tasks in a simulated household environment. Our VirtualHome simulator allows us to create a large activity video dataset with rich ground-truth, enabling training and testing of video understanding models. We further showcase examples of our agent performing tasks in our VirtualHome based on language descriptions.
Code (4)
Tasks
Video UnderstandingSimilar Papers 제목 키워드 기반
Synthesizing Environment-Aware Activities via Activity Sketches
In order to learn to perform activities from demonstrations or descriptions, agents need to distill what the essence of the given activity is, and how it can be adapted to new environments. In this work, we address the p…
Gaze-Guided Graph Neural Network for Action Anticipation Conditioned on Intention
Humans utilize their gaze to concentrate on essential information while perceiving and interpreting intentions in videos. Incorporating human gaze into computational algorithms can significantly enhance model performance…
Action AnticipationGraph Neural NetworkIntent DetectionVideo UnderstandingProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Task planning can require defining myriad domain knowledge about the world in which a robot needs to act. To ameliorate that effort, large language models (LLMs) can be used to score potential next actions during task pl…
Task PlanningHybrid Top-Down and Bottom-Up Approach for Investigating Residential Load Compositions and Load Percentages
Load points are one of the most vital parts of power systems. Due to the new load forms and programs introduced in the demand side, the load-serving entities (LSEs) no longer deal with lump loads, but rather with more dy…
ManagementSynthesizing Event-centric Knowledge Graphs of Daily Activities Using Virtual Space
Artificial intelligence (AI) is expected to be embodied in software agents, robots, and cyber-physical systems that can understand the various contextual information of daily life in the home environment to support human…
Decision MakingEmbodied Question AnsweringKnowledge GraphsQuestion Answering