Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observation
We propose a learning approach for mapping context-dependent sequential instructions to actions. We address the problem of discourse and state dependencies with an attention-based model that considers both the history of the interaction and the state of the world. To train from start and goal states without access to demonstrations, we propose SESTRA, a learning algorithm that takes advantage of single-step reward observations and immediate expected reward maximization. We evaluate on the SCONE domains, and show absolute accuracy improvements of 9.8%-25.3% across the domains over approaches that use high-level logical representations.
Code (1)
Similar Papers 제목 키워드 기반
Situated Instruction Following
Language is never spoken in a vacuum. It is expressed, comprehended, and contextualized within the holistic backdrop of the speaker's history, actions, and environment. Since humans are used to communicating efficiently …
Instruction FollowingWeakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions
The context in which language is used provides a strong signal for learning to recover its meaning. In this paper, we show it can be used within a grounded CCG semantic parsing approach that learns a joint model of meani…
Semantic ParsingWeakly-supervised LearningLanguage-Guided Adaptive Perception for Efficient Grounded Communication with Robotic Manipulators in Cluttered Environments
The utility of collaborative manipulators for shared tasks is highly dependent on the speed and accuracy of communication between the human and the robot. The run-time of recently developed probabilistic inference models…
Robot ManipulationA Computational Model for Situated Task Learning with Interactive Instruction
Learning novel tasks is a complex cognitive activity requiring the learner to acquire diverse declarative and procedural knowledge. Prior ACT-R models of acquiring task knowledge from instruction focused on learning proc…
Executing Instructions in Situated Collaborative Interactions
We study a collaborative scenario where a user not only instructs a system to complete tasks, but also acts alongside it. This allows the user to adapt to the system abilities by changing their language or deciding to si…