The Boombox: Visual Reconstruction from Acoustic Vibrations
Interacting with bins and containers is a fundamental task in robotics, making state estimation of the objects inside the bin critical. While robots often use cameras for state estimation, the visual modality is not always ideal due to occlusions and poor illumination. We introduce The Boombox, a container that uses sound to estimate the state of the contents inside a box. Based on the observation that the collision between objects and its containers will cause an acoustic vibration, we present a convolutional network for learning to reconstruct visual scenes. Although we use low-cost and low-power contact microphones to detect the vibrations, our results show that learning from multimodal data enables state estimation from affordable audio sensors. Due to the many ways that robots use containers, we believe the box will have a number of applications in robotics. Our project website is at: boombox.cs.columbia.edu
Code (1)
Tasks
State EstimationSimilar Papers 제목 키워드 기반
A Speech Production Model for Radar: Connecting Speech Acoustics with Radar-Measured Vibrations
Millimeter Wave (mmWave) radar has emerged as a promising modality for speech sensing, offering advantages over traditional microphones. Prior works have demonstrated that radar captures motion signals related to vocal v…
Speech EnhancementEvent2Audio: Event-Based Optical Vibration Sensing
Small vibrations observed in video can unveil information beyond what is visual, such as sound and material properties. It is possible to passively record these vibrations when they are visually perceptible, or actively …
Dual-Shutter Optical Vibration Sensing
Visual vibrometry is a highly useful tool for remote capture of audio, as well as the physical properties of materials, human heart rate, and more. While visually-observable vibrations can be captured directly with a…
Condition Monitoring of Drive Trains by Data Fusion of Acoustic Emission and Vibration Sensors
Early damage detection and classification by condition monitoring systems is crucial to enable predictive maintenance of manufacturing systems and industrial facilities. Data analysis can be improved by applying machine …
ClassificationFault DetectionTime Series AnalysisMicro-Vibration Modes Reconstruction Based on Micro-Doppler Coincidence Imaging
Micro-vibration, a ubiquitous nature phenomenon, can be seen as a characteristic feature on the objects, these vibrations always have tiny amplitudes which are much less than the wavelengths of the sensing systems, thus …