Nebula: A discourse aware Minecraft Builder
When engaging in collaborative tasks, humans efficiently exploit the semantic structure of a conversation to optimize verbal and nonverbal interactions. But in recent "language to code" or "language to action" models, this information is lacking. We show how incorporating the prior discourse and nonlinguistic context of a conversation situated in a nonlinguistic environment can improve the "language to action" component of such interactions. We finetune an LLM to predict actions based on prior context; our model, Nebula, doubles the net-action F1 score over the baseline on this task of Jayannavar et al.(2020). We also investigate our model's ability to construct shapes and understand location descriptions using a synthetic dataset
Code (0)
등록된 구현이 없습니다.
Tasks
MinecraftSimilar Papers 제목 키워드 기반
A LLM Benchmark based on the Minecraft Builder Dialog Agent Task
In this work we proposing adapting the Minecraft builder task into an LLM benchmark suitable for evaluating LLM ability in spatially orientated tasks, and informing builder agent design. Previous works have proposed corp…
MathMinecraftSpatial ReasoningRetrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft
In the Minecraft Collaborative Building Task, two players collaborate: an Architect (A) provides instructions to a Builder (B) to assemble a specified structure using 3D blocks. In this work, we investigate the use of la…
Action GenerationCode GenerationIn-Context LearningMinecraft+1Learning to Execute Actions or Ask Clarification Questions
Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. We wish to develop an intelligent builder agent in a si…
Learning to ExecuteMinecraftLearning to execute or ask clarification questions
Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. To this end, we wish to develop an intelligent builder …
Learning to ExecuteMinecraftLarge Language Models as Minecraft Agents
In this work we examine the use of Large Language Models (LLMs) in the challenging setting of acting as a Minecraft agent. We apply and evaluate LLMs in the builder and architect settings, introduce clarification questio…
Minecraft