paper-with-me

홈 › Papers

Learning Spatiotemporal Frequency-Transformer for Compressed Video Super-Resolution

2022-08-05 · Zhongwei Qiu, Huan Yang, Jianlong Fu, Dongmei Fu

Compressed video super-resolution (VSR) aims to restore high-resolution frames from compressed low-resolution counterparts. Most recent VSR approaches often enhance an input frame by borrowing relevant textures from neighboring video frames. Although some progress has been made, there are grand challenges to effectively extract and transfer high-quality textures from compressed videos where most frames are usually highly degraded. In this paper, we propose a novel Frequency-Transformer for compressed video super-resolution (FTVSR) that conducts self-attention over a joint space-time-frequency domain. First, we divide a video frame into patches, and transform each patch into DCT spectral maps in which each channel represents a frequency band. Such a design enables a fine-grained level self-attention on each frequency band, so that real visual texture can be distinguished from artifacts, and further utilized for video frame restoration. Second, we study different self-attention schemes, and discover that a divided attention which conducts a joint space-frequency attention before applying temporal attention on each frequency band, leads to the best video enhancement quality. Experimental results on two widely-used video super-resolution benchmarks show that FTVSR outperforms state-of-the-art approaches on both uncompressed and compressed videos with clear visual margins. Code is available at https://github.com/researchmm/FTVSR.

📄 PDF Abstract BibTeX arXiv:2208.03012

Code (1)

researchmm/ftvsr 공식 구현 pytorch

Tasks

Super-ResolutionVideo EnhancementVideo Super-Resolution

Similar Papers 제목 키워드 기반

Frequency-Aware Spatiotemporal Transformers for Video Inpainting Detection

2021-01-01 · ICCV 2021 10 · Bingyao Yu, Wanhua Li, Xiu Li, Jiwen Lu 외

In this paper, we propose a frequency-aware spatiotemporal transformers for deep In this paper, we propose a Frequency-Aware Spatiotemporal Transformer (FAST) for video inpainting detection, which aims to simultaneou…

DecoderVideo Inpainting

Compressed Deepfake Video Detection Based on 3D Spatiotemporal Trajectories

2024-04-28 · Zongmei Chen, Xin Liao, Xiaoshuai Wu, Yanxiang Chen

The misuse of deepfake technology by malicious actors poses a potential threat to nations, societies, and individuals. However, existing methods for detecting deepfakes primarily focus on uncompressed videos, such as noi…

Face Swapping

MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition

2021-08-20 · Jiawei Chen, Chiu Man Ho

This paper presents a pure transformer-based approach, dubbed the Multi-Modal Video Transformer (MM-ViT), for video action recognition. Different from other schemes which solely utilize the decoded RGB frames, MM-ViT ope…

Action RecognitionOptical Flow EstimationTemporal Action Localization

Learning Spatiotemporal Frequency-Transformer for Low-Quality Video Super-Resolution

2022-12-27 · Zhongwei Qiu, Huan Yang, Jianlong Fu, Daochang Liu 외

Video Super-Resolution (VSR) aims to restore high-resolution (HR) videos from low-resolution (LR) videos. Existing VSR techniques usually recover HR frames by extracting pertinent textures from nearby frames with known d…

Super-ResolutionVideo EnhancementVideo Super-Resolution

Self-supervised Video Transformer

2021-12-02 · CVPR 2022 1 · Kanchana Ranasinghe, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan 외

In this paper, we propose self-supervised training for video transformers using unlabeled video data. From a given video, we create local and global spatiotemporal views with varying spatial sizes and frame rates. Our se…

Action ClassificationAction RecognitionAction Recognition In VideosSelf-Supervised Action Recognition Linear