Unsupervised uncertainty estimation using spatiotemporal cues in video saliency detection
In this paper, we address the problem of quantifying reliability of computational saliency for videos, which can be used to improve saliency-based video processing and enable more reliable performance and risk assessment of such processing. Our approach is twofold. First, we explore spatial correlations in both saliency map and eye-fixation map. Then, we learn spatiotemporal correlations that define a reliable saliency map. We first study spatiotemporal eye-fixation data from a public dataset and investigate a common feature in human visual attention, which dictates correlation in saliency between a pixel and its direct neighbors. Based on the study, we then develop an algorithm that estimates a pixel-wise uncertainty map that reflects our confidence in the associated computational saliency map by relating a pixel's saliency to the saliency of its neighbors. To estimate such uncertainties, we measure the divergence of a pixel, in a saliency map, from its local neighborhood. Additionally, we propose a systematic procedure to evaluate the estimation performance by explicitly computing uncertainty ground truth as a function of a given saliency map and eye fixations of human subjects. In our experiments, we explore multiple definitions of locality and neighborhoods in spatiotemporal video signals. In addition, we examine the relationship between the parameters of our proposed algorithm and the content of the videos. The proposed algorithm is unsupervised, making it more suitable for generalization to most natural videos. Also, it is computationally efficient and flexible for customization to specific video content. Experiments using three publicly available video datasets show that the proposed algorithm outperforms state-of-the-art uncertainty estimation methods with improvement in accuracy up to 63% and offers efficiency and flexibility that make it more useful in practical situations.
Code (0)
등록된 구현이 없습니다.
Tasks
Saliency DetectionVideo Saliency DetectionSimilar Papers 제목 키워드 기반
A Benchmark Dataset and Saliency-guided Stacked Autoencoders for Video-based Salient Object Detection
Image-based salient object detection (SOD) has been extensively studied in the past decades. However, video-based SOD is much less explored since there lack large-scale video datasets within which salient objects are una…
BenchmarkingObjectobject-detectionObject Detection+2SalSum: Saliency-based Video Summarization using Generative Adversarial Networks
The huge amount of video data produced daily by camera-based systems, such as surveilance, medical and telecommunication systems, emerges the need for effective video summarization (VS) methods. These methods should be c…
Generative Adversarial NetworkVideo SummarizationLearning Temporally and Semantically Consistent Unpaired Video-to-video Translation Through Pseudo-Supervision From Synthetic Optical Flow
Unpaired video-to-video translation aims to translate videos between a source and a target domain without the need of paired training data, making it more feasible for real applications. Unfortunately, the translated vid…
Motion EstimationOptical Flow EstimationTranslationDeepfake Video Detection with Spatiotemporal Dropout Transformer
While the abuse of deepfake technology has caused serious concerns recently, how to detect deepfake videos is still a challenge due to the high photo-realistic synthesis of each frame. Existing image-level approaches oft…
Data AugmentationFace SwappingSpatiotemporal Contrastive Video Representation Learning
We present a self-supervised Contrastive Video Representation Learning (CVRL) method to learn spatiotemporal visual representations from unlabeled videos. Our representations are learned using a contrastive loss, where t…
Action RecognitionContrastive LearningData AugmentationRepresentation Learning+4