Executing Instructions in Situated Collaborative Interactions
We study a collaborative scenario where a user not only instructs a system to complete tasks, but also acts alongside it. This allows the user to adapt to the system abilities by changing their language or deciding to simply accomplish some tasks themselves, and requires the system to effectively recover from errors as the user strategically assigns it new goals. We build a game environment to study this scenario, and learn to map user instructions to system actions. We introduce a learning approach focused on recovery from cascading errors between instructions, and modeling methods to explicitly reason about instructions with multiple goals. We evaluate with a new evaluation protocol using recorded interactions and online games with human users, and observe how users adapt to the system abilities.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions
The context in which language is used provides a strong signal for learning to recover its meaning. In this paper, we show it can be used within a grounded CCG semantic parsing approach that learns a joint model of meani…
Semantic ParsingWeakly-supervised LearningLanguage-Guided Adaptive Perception for Efficient Grounded Communication with Robotic Manipulators in Cluttered Environments
The utility of collaborative manipulators for shared tasks is highly dependent on the speed and accuracy of communication between the human and the robot. The run-time of recently developed probabilistic inference models…
Robot ManipulationMindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks
An ideal integration of autonomous agents in a human world implies that they are able to collaborate on human terms. In particular, theory of mind plays an important role in maintaining common ground during human collabo…
MinecraftTheory of Mind ModelingA Computational Model for Situated Task Learning with Interactive Instruction
Learning novel tasks is a complex cognitive activity requiring the learner to acquire diverse declarative and procedural knowledge. Prior ACT-R models of acquiring task knowledge from instruction focused on learning proc…
How Should Agents Ask Questions For Situated Learning? An Annotated Dialogue Corpus
Intelligent agents that are confronted with novel concepts in situated environments will need to ask their human teammates questions to learn about the physical world. To better understand this problem, we need data abou…
Novel ConceptsQuestion GenerationQuestion-Generation