Diagnosing and Preventing Instabilities in Recurrent Video Processing
Recurrent models are a popular choice for video enhancement tasks such as video denoising or super-resolution. In this work, we focus on their stability as dynamical systems and show that they tend to fail catastrophically at inference time on long video sequences. To address this issue, we (1) introduce a diagnostic tool which produces input sequences optimized to trigger instabilities and that can be interpreted as visualizations of temporal receptive fields, and (2) propose two approaches to enforce the stability of a model during training: constraining the spectral norm or constraining the stable rank of its convolutional layers. We then introduce Stable Rank Normalization for Convolutional layers (SRN-C), a new algorithm that enforces these constraints. Our experimental results suggest that SRN-C successfully enforces stability in recurrent video processing models without a significant performance loss.
Code (0)
등록된 구현이 없습니다.
Tasks
DenoisingDiagnosticSuper-ResolutionVideo DenoisingVideo EnhancementSimilar Papers 제목 키워드 기반
Stable Long-Term Recurrent Video Super-Resolution
Recurrent models have gained popularity in deep learning (DL) based video super-resolution (VSR), due to their increased computational efficiency, temporal receptive field and temporal consistency compared to sliding-win…
Computational EfficiencySuper-ResolutionVideo Super-ResolutionLearning to navigate image manifolds induced by generative adversarial networks for unsupervised video generation
In this work, we introduce a two-step framework for generative modeling of temporal data. Specifically, the generative adversarial networks (GANs) setting is employed to generate synthetic scenes of moving objects. To do…
NavigateVideo GenerationRecurrent Convolutions for Causal 3D CNNs
Recently, three dimensional (3D) convolutional neural networks (CNNs) have emerged as dominant methods to capture spatiotemporal representations in videos, by adding to pre-existing 2D CNNs a third, temporal dimension. S…
Action DetectionThe Dynamics of Signal Propagation in Gated Recurrent Neural Networks
Training recurrent neural networks (RNNs) on long sequence tasks is plagued with difficulties arising from the exponential explosion or vanishing of signals as they propagate forward or backward through the network. Many…
Higher Order Recurrent Space-Time Transformer for Video Action Prediction
Endowing visual agents with predictive capability is a key step towards video intelligence at scale. The predominant modeling paradigm for this is sequence learning, mostly implemented through LSTMs. Feed-forward Transfo…
Action AnticipationAction RecognitionAttribute