RMM: A Recursive Mental Model for Dialogue Navigation
Language-guided robots must be able to both ask humans questions and understand answers. Much existing work focuses only on the latter. In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers. Inspired by theory of mind, we propose the Recursive Mental Model (RMM). The navigating agent models the guiding agent to simulate answers given candidate generated questions. The guiding agent in turn models the navigating agent to simulate navigation steps it would take to generate answers. We use the progress agents make towards the goal as a reinforcement learning reward signal to directly inform not only navigation actions, but also both question and answer generation. We demonstrate that RMM enables better generalization to novel environments. Interlocutor modelling may be a way forward for human-agent RMM where robots need to both ask and answer questions.
Code (1)
Tasks
Answer GenerationInstruction FollowingmodelSimilar Papers 제목 키워드 기반
RMM: A Recursive Mental Model for Dialog Navigation
Language-guided robots must be able to both ask humans questions and understand answers. Much existing work focuses only on the latter. In this paper, we go beyond instruction following and introduce a two-agent task whe…
Answer GenerationInstruction FollowingmodelUtterance Intent Classification of a Spoken Dialogue System with Efficiently Untied Recursive Autoencoders
Recursive autoencoders (RAEs) for compositionality of a vector space model were applied to utterance intent classification of a smartphone-based Japanese-language spoken dialogue system. Though the RAEs express a nonline…
Automatic Speech Recognition (ASR)ClassificationGeneral Classificationintent-classification+4Recursive Visual Imagination and Adaptive Linguistic Grounding for Vision Language Navigation
Vision Language Navigation (VLN) typically requires agents to navigate to specified objects or remote regions in unknown scenes by obeying linguistic commands. Such tasks require organizing historical visual observations…
Information Navigation System with Discovering User Interests
We demonstrate an information navigation system for sightseeing domains that has a dialogue interface for discovering user interests for tourist activities. The system discovers interests of a user with focus detection o…
Semantic Textual SimilaritySpeech RecognitionNDH-Full: Learning and Evaluating Navigational Agents on Full-Length Dialogue
Communication between human and mobile agents is getting increasingly important as such agents are widely deployed in our daily lives. Vision-and-Dialogue Navigation is one of the tasks that evaluate the agent’s ability …
Data AugmentationDynamic Time WarpingNavigate