Interpretability in Action: Exploratory Analysis of VPT, a Minecraft Agent
Understanding the mechanisms behind decisions taken by large foundation models in sequential decision making tasks is critical to ensuring that such systems operate transparently and safely. In this work, we perform exploratory analysis on the Video PreTraining (VPT) Minecraft playing agent, one of the largest open-source vision-based agents. We aim to illuminate its reasoning mechanisms by applying various interpretability techniques. First, we analyze the attention mechanism while the agent solves its training task - crafting a diamond pickaxe. The agent pays attention to the last four frames and several key-frames further back in its six-second memory. This is a possible mechanism for maintaining coherence in a task that takes 3-10 minutes, despite the short memory span. Secondly, we perform various interventions, which help us uncover a worrying case of goal misgeneralization: VPT mistakenly identifies a villager wearing brown clothes as a tree trunk when the villager is positioned stationary under green tree leaves, and punches it to death.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingMinecraftSequential Decision MakingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ADAM: An Embodied Causal Agent in Open-World Environments
In open-world environments like Minecraft, existing agents face challenges in continuously learning structured knowledge, particularly causality. These challenges stem from the opacity inherent in black-box models and an…
Lifelong learningMinecraftNavigateWorld KnowledgeOpen-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction
We study the problem of learning goal-conditioned policies in Minecraft, a popular, widely accessible yet challenging open-ended environment for developing human-level multi-task agents. We first identify two main challe…
DiversityMinecraftRepresentation LearningZero-shot GeneralizationOptimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy
Building an agent that can mimic human behavior patterns to accomplish various open-world tasks is a long-term goal. To enable agents to effectively learn behavioral patterns across diverse tasks, a key challenge lies in…
Large Language ModelMinecraftMultimodal Large Language ModelDescribe, Explain, Plan and Select: Interactive Planning with LLMs Enables Open-World Multi-Task Agents
In this paper, we study the problem of planning in Minecraft, a popular, democratized yet challenging open-ended environment for developing multi-task embodied agents. We've found two primary challenges of empowering suc…
Enabling Rapid Shared Human-AI Mental Model Alignment via the After-Action Review
In this work, we present two novel contributions toward improving research in human-machine teaming (HMT): 1) a Minecraft testbed to accelerate testing and deployment of collaborative AI agents and 2) a tool to allow use…
Minecraft