Combining Vision and Tactile Sensation for Video Prediction
In this paper, we explore the impact of adding tactile sensation to video prediction models for physical robot interactions. Predicting the impact of robotic actions on the environment is a fundamental challenge in robotics. Current methods leverage visual and robot action data to generate video predictions over a given time period, which can then be used to adjust robot actions. However, humans rely on both visual and tactile feedback to develop and maintain a mental model of their physical surroundings. In this paper, we investigate the impact of integrating tactile feedback into video prediction models for physical robot interactions. We propose three multi-modal integration approaches and compare the performance of these tactile-enhanced video prediction models. Additionally, we introduce two new datasets of robot pushing that use a magnetic-based tactile sensor for unsupervised learning. The first dataset contains visually identical objects with different physical properties, while the second dataset mimics existing robot-pushing datasets of household object clusters. Our results demonstrate that incorporating tactile feedback into video prediction models improves scene prediction accuracy and enhances the agent's perception of physical interactions and understanding of cause-effect relationships during physical robot interactions.
Code (0)
등록된 구현이 없습니다.
Tasks
PredictionVideo PredictionSimilar Papers 제목 키워드 기반
FlowTouch: View-Invariant Visuo-Tactile Prediction
Tactile sensation is essential for contact-rich manipulation tasks. It provides direct feedback on object geometry, surface properties, and interaction forces, enhancing perception and enabling fine-grained control. An i…
Image GenerationTelextiles: End-to-end Remote Transmission of Fabric Tactile Sensation
The tactile sensation of textiles is critical in determining the comfort of clothing. For remote use, such as online shopping, users cannot physically touch the textile of clothes, making it difficult to evaluate its tac…
Self-Supervised LearningTextToucher: Fine-Grained Text-to-Touch Generation
Tactile sensation plays a crucial role in the development of multi-modal large models and embodied intelligence. To collect tactile data with minimal cost as possible, a series of studies have attempted to generate tacti…
Language ModellingLarge Language ModelMultimodal Large Language ModelTactile Data Recording System for Clothing with Motion-Controlled Robotic Sliding
The tactile sensation of clothing is critical to wearer comfort. To reveal physical properties that make clothing comfortable, systematic collection of tactile data during sliding motion is required. We propose a robotic…
Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training
The rapidly evolving field of robotics necessitates methods that can facilitate the fusion of multiple modalities. Specifically, when it comes to interacting with tangible objects, effectively combining visual and tactil…
Contrastive LearningMaterial ClassificationRepresentation Learning