Exploring the Mutual Influence between Self-Supervised Single-Frame and Multi-Frame Depth Estimation
Although both self-supervised single-frame and multi-frame depth estimation methods only require unlabeled monocular videos for training, the information they leverage varies because single-frame methods mainly rely on appearance-based features while multi-frame methods focus on geometric cues. Considering the complementary information of single-frame and multi-frame methods, some works attempt to leverage single-frame depth to improve multi-frame depth. However, these methods can neither exploit the difference between single-frame depth and multi-frame depth to improve multi-frame depth nor leverage multi-frame depth to optimize single-frame depth models. To fully utilize the mutual influence between single-frame and multi-frame methods, we propose a novel self-supervised training framework. Specifically, we first introduce a pixel-wise adaptive depth sampling module guided by single-frame depth to train the multi-frame model. Then, we leverage the minimum reprojection based distillation loss to transfer the knowledge from the multi-frame depth network to the single-frame network to improve single-frame depth. Finally, we regard the improved single-frame depth as a prior to further boost the performance of multi-frame depth estimation. Experimental results on the KITTI and Cityscapes datasets show that our method outperforms existing approaches in the self-supervised monocular setting.
Code (1)
Tasks
Depth EstimationSimilar Papers 제목 키워드 기반
Robust Mutual Learning for Semi-supervised Semantic Segmentation
Recent semi-supervised learning (SSL) methods are commonly based on pseudo labeling. Since the SSL performance is greatly influenced by the quality of pseudo labels, mutual learning has been proposed to effectively suppr…
Pseudo LabelSemantic SegmentationSemi-Supervised Semantic SegmentationAn Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settings
There is a growing need for pluralistic alignment methods that can steer language models towards individual attributes and preferences. One such method, Self-Supervised Alignment with Mutual Information (SAMI), uses cond…
8kMathGeneralizing Supervised Contrastive learning: A Projection Perspective
Self-supervised contrastive learning (SSCL) has emerged as a powerful paradigm for representation learning and has been studied from multiple perspectives, including mutual information and geometric viewpoints. However, …
Contrastive LearningRepresentation LearningvalidLearning Representations by Maximizing Mutual Information Across Views
We propose an approach to self-supervised representation learning based on maximizing mutual information between features extracted from multiple views of a shared context. For example, one could produce multiple views o…
Data AugmentationImage ClassificationLinear evaluationRepresentation Learning+2Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective
Existing studies on self-supervised speech representation learning have focused on developing new training methods and applying pre-trained models for different applications. However, the quality of these models is often…
Representation LearningSelf-Supervised Learningspeech-recognitionSpeech Recognition+1