Joint Object-Material Category Segmentation from Audio-Visual Cues
It is not always possible to recognise objects and infer material properties for a scene from visual cues alone, since objects can look visually similar whilst being made of very different materials. In this paper, we therefore present an approach that augments the available dense visual cues with sparse auditory cues in order to estimate dense object and material labels. Since estimates of object class and material properties are mutually informative, we optimise our multi-output labelling jointly using a random-field framework. We evaluate our system on a new dataset with paired visual and auditory data that we make publicly available. We demonstrate that this joint estimation of object and material labels significantly outperforms the estimation of either category in isolation.
Code (0)
등록된 구현이 없습니다.
Tasks
ObjectSimilar Papers 제목 키워드 기반
Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics
The audio-visual segmentation (AVS) task aims to segment sounding objects from a given video. Existing works mainly focus on fusing audio and visual features of a given video to achieve sounding object masks. However, we…
ObjectSegmentationSemantic SegmentationTransavs: End-To-End Audio-Visual Segmentation With Transformer
Audio-Visual Segmentation (AVS) is a challenging task, which aims to segment sounding objects in video frames by exploring audio signals. Generally AVS faces two key challenges: (1) Audio signals inherently exhibit a hig…
Scene UnderstandingSegmentationSemantic SegmentationAudio-Visual Segmentation with Semantics
We propose a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame. To facilitate this research, we constr…
SegmentationSemantic SegmentationVideo Semantic SegmentationCategory-Level Articulated Object Pose Estimation
This project addresses the task of category-level pose estimation for articulated objects from a single depth image. We present a novel category-level approach that correctly accommodates object instances previously unse…
Objectparameter estimationPose EstimationIntegrating Local Material Recognition with Large-Scale Perceptual Attribute Discovery
Material attributes have been shown to provide a discriminative intermediate representation for recognizing materials, especially for the challenging task of recognition from local material appearance (i.e., regardless o…
AttributeMaterial Recognition