Audio-Visual Object Classification for Human-Robot Collaboration
Human-robot collaboration requires the contactless estimation of the physical properties of containers manipulated by a person, for example while pouring content in a cup or moving a food box. Acoustic and visual signals can be used to estimate the physical properties of such objects, which may vary substantially in shape, material and size, and also be occluded by the hands of the person. To facilitate comparisons and stimulate progress in solving this problem, we present the CORSMAL challenge and a dataset to assess the performance of the algorithms through a set of well-defined performance scores. The tasks of the challenge are the estimation of the mass, capacity, and dimensions of the object (container), and the classification of the type and amount of its content. A novel feature of the challenge is our real-to-simulation framework for visualising and assessing the impact of estimation errors in human-to-robot handovers.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationObjectSimilar Papers 제목 키워드 기반
CAVER: Curious Audiovisual Exploring Robot
Multimodal audiovisual perception can enable new avenues for robotic manipulation, from better material classification to the imitation of demonstrations for which only audio signals are available (e.g., playing a tune b…
Deep Neural Object Analysis by Interactive Auditory Exploration with a Humanoid Robot
We present a novel approach for interactive auditory object analysis with a humanoid robot. The robot elicits sensory information by physically shaking visually indistinguishable plastic capsules. It gathers the resultin…
General ClassificationMaterial ClassificationManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data
Audio signals provide rich information for the robot interaction and object properties through contact. This information can surprisingly ease the learning of contact-rich robot manipulation skills, especially when the v…
Contact-rich ManipulationObjectRobot ManipulationStyle-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos
To realize human-robot collaboration, robots need to execute actions for new tasks according to human instructions given finite prior knowledge. Human experts can share their knowledge of how to perform a task with a rob…
Multi-Task LearningScene UnderstandingStyle TransferVideo Captioning+1A proto-object based audiovisual saliency map
Natural environment and our interaction with it is essentially multisensory, where we may deploy visual, tactile and/or auditory senses to perceive, learn and interact with our environment. Our objective in this study is…
ObjectvalidVideo Compression