A LLM Benchmark based on the Minecraft Builder Dialog Agent Task
In this work we proposing adapting the Minecraft builder task into an LLM benchmark suitable for evaluating LLM ability in spatially orientated tasks, and informing builder agent design. Previous works have proposed corpora with varying complex structures, and human written instructions. We instead attempt to provide a comprehensive synthetic benchmark for testing builder agents over a series of distinct tasks that comprise of common building operations. We believe this approach allows us to probe specific strengths and weaknesses of different agents, and test the ability of LLMs in the challenging area of spatial reasoning and vector based math.
Code (0)
등록된 구현이 없습니다.
Tasks
MathMinecraftSpatial ReasoningSimilar Papers 제목 키워드 기반
Collaborative Dialogue in Minecraft
We wish to develop interactive agents that can communicate with humans to collaboratively solve tasks in grounded scenarios. Since computer games allow us to simulate such tasks without the need for physical robots, we d…
MinecraftBAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues
Interactive agents capable of understanding and executing instructions in the physical world have long been a central goal in AI research. The Minecraft Collaborative Building Task (MCBT) provides one such setting to wor…
Instruction FollowingMinecraftLearning to Execute Actions or Ask Clarification Questions
Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. We wish to develop an intelligent builder agent in a si…
Learning to ExecuteMinecraftLearning to execute or ask clarification questions
Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. To this end, we wish to develop an intelligent builder …
Learning to ExecuteMinecraftLearning to execute instructions in a Minecraft dialogue
The Minecraft Collaborative Building Task is a two-player game in which an Architect (A) instructs a Builder (B) to construct a target structure in a simulated Blocks World Environment. We define the subtask of predictin…
Learning to ExecuteMinecraft