GROOT: Learning to Follow Instructions by Watching Gameplay Videos

2023-10-12 · Shaofei Cai, Bowei Zhang, ZiHao Wang, Xiaojian Ma, Anji Liu, Yitao Liang

We study the problem of building a controller that can follow open-ended instructions in open-world environments. We propose to follow reference videos as instructions, which offer expressive goal specifications while eliminating the need for expensive text-gameplay annotations. A new learning framework is derived to allow learning such instruction-following controllers from gameplay videos while producing a video instruction encoder that induces a structured goal space. We implement our agent GROOT in a simple yet effective encoder-decoder architecture based on causal transformers. We evaluate GROOT against open-world counterparts and human players on a proposed Minecraft SkillForge benchmark. The Elo ratings clearly show that GROOT is closing the human-machine gap as well as exhibiting a 70% winning rate over the best generalist agent baseline. Qualitative analysis of the induced goal space further demonstrates some interesting emergent properties, including the goal composition and complex gameplay behavior synthesis. The project page is available at https://craftjarvis-groot.github.io.

📄 PDF Abstract BibTeX arXiv:2310.08235

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder Instruction FollowingMinecraft

GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agents

2024-12-07 · Shaofei Cai, Bowei Zhang, ZiHao Wang, Haowei Lin 외

Developing agents that can follow multimodal instructions remains a fundamental challenge in robotics and AI. Although large-scale pre-training on unlabeled datasets (no language instruction) has enabled agents to learn …

Instruction Following

MarioQA: Answering Questions by Watching Gameplay Videos

2016-12-06 · ICCV 2017 10 · Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, Bohyung Han

We present a framework to analyze various aspects of models for video question answering (VideoQA) using customizable synthetic datasets, which are constructed automatically from gameplay videos. Our work is motivated by…

Question Answering Video Question Answering

CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning

2022-03-21 · Mohammad Reza Taesiri, Finlay Macklon, Cor-Paul Bezemer

Gameplay videos contain rich information about how players interact with the game and how the game responds. Sharing gameplay videos on social media platforms, such as Reddit, has become a common practice for many player…

Event Detection Transfer Learning

Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos

2022-06-23 · Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga 외

Pretraining on noisy, internet-scale datasets has been heavily studied as a technique for training models with broad, general capabilities for text, images, and other modalities. However, for many sequential decision dom…

Imitation Learning Minecraft reinforcement-learning Reinforcement Learning (RL)

Learning Generalizable Manipulation Policies with Object-Centric 3D Representations

2023-10-22 · Yifeng Zhu, Zhenyu Jiang, Peter Stone, Yuke Zhu

We introduce GROOT, an imitation learning method for learning robust policies with object-centric and 3D priors. GROOT builds policies that generalize beyond their initial training conditions for vision-based manipulatio…

Imitation Learning Object