Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations
Aging societies, labor shortages and increasing wage costs call for assistance robots capable of autonomously performing a wide array of real-world tasks. Such open-ended robotic manipulation requires not only powerful knowledge representations and reasoning (KR&R) algorithms, but also methods for humans to instruct robots what tasks to perform and how to perform them. In this paper, we present a system for automatically generating executable robot control programs from human task demonstrations in virtual reality (VR). We leverage common-sense knowledge and game engine-based physics to semantically interpret human VR demonstrations, as well as an expressive and general task representation and automatic path planning and code generation, embedded into a state-of-the-art cognitive architecture. We demonstrate our approach in the context of force-sensitive fetch-and-place for a robotic shopping assistant. The source code is available at https://github.com/ease-crc/vr-program-synthesis.
Code (2)
Tasks
Code GenerationCommon Sense ReasoningProgram SynthesisSimilar Papers 제목 키워드 기반
Knowledge-Driven Program Synthesis via Adaptive Replacement Mutation and Auto-constructed Subprogram Archives
We introduce Knowledge-Driven Program Synthesis (KDPS) as a variant of the program synthesis task that requires the agent to solve a sequence of program synthesis problems. In KDPS, the agent should use knowledge from th…
Program SynthesisBF++: a language for general-purpose program synthesis
Most state of the art decision systems based on Reinforcement Learning (RL) are data-driven black-box neural models, where it is often difficult to incorporate expert knowledge into the models or let experts review and v…
Decision MakingOpenAI GymProgram SynthesisReinforcement Learning (RL)GALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesis
Despite achieving superior performance in human-level control problems, unlike humans, deep reinforcement learning (DRL) lacks high-order intelligence (e.g., logic deduction and reuse), thus it behaves ineffectively than…
Decision MakingDeep Reinforcement LearningProgram Synthesisreinforcement-learning+2BANSAI: Towards Bridging the AI Adoption Gap in Industrial Robotics with Neurosymbolic Programming
Over the past decade, deep learning helped solve manipulation problems across all domains of robotics. At the same time, industrial robots continue to be programmed overwhelmingly using traditional program representation…
Industrial RobotsProgram SynthesisA Divide-Align-Conquer Strategy for Program Synthesis
A major bottleneck in search-based program synthesis is the exponentially growing search space which makes learning large programs intractable. Humans mitigate this problem by leveraging the compositional nature of the r…
ARCInductive logic programmingProgram SynthesisVisual Reasoning