Unsupervised Video-to-Video Translation
Unsupervised image-to-image translation is a recently proposed task of translating an image to a different style or domain given only unpaired image examples at training time. In this paper, we formulate a new task of unsupervised video-to-video translation, which poses its own unique challenges. Translating video implies learning not only the appearance of objects and scenes but also realistic motion and transitions between consecutive frames.We investigate the performance of per-frame video-to-video translation using existing image-to-image translation networks, and propose a spatio-temporal 3D translator as an alternative solution to this problem. We evaluate our 3D method on multiple synthetic datasets, such as moving colorized digits, as well as the realistic segmentation-to-video GTA dataset and a new CT-to-MRI volumetric images translation dataset. Our results show that frame-wise translation produces realistic results on a single frame level but underperforms significantly on the scale of the whole video compared to our three-dimensional translation approach, which is better able to learn the complex structure of video and motion and continuity of object appearance.
Code (1)
Tasks
Image-to-Image TranslationTranslationUnsupervised Image-To-Image TranslationSimilar Papers 제목 키워드 기반
Unsupervised Multimodal Video-to-Video Translation via Self-Supervised Learning
Existing unsupervised video-to-video translation methods fail to produce translated videos which are frame-wise realistic, semantic information preserving and video-level consistent. In this work, we propose UVIT, a nove…
DecoderSelf-Supervised LearningTranslationUnsupervised Sign Language Translation and Generation
Motivated by the success of unsupervised neural machine translation (UNMT), we introduce an unsupervised sign language translation and generation network (USLNet), which learns from abundant single-modality (text and vid…
Machine TranslationSign Language TranslationTranslationLearning Temporally and Semantically Consistent Unpaired Video-to-video Translation Through Pseudo-Supervision From Synthetic Optical Flow
Unpaired video-to-video translation aims to translate videos between a source and a target domain without the need of paired training data, making it more feasible for real applications. Unfortunately, the translated vid…
Motion EstimationOptical Flow EstimationTranslationVHS to HDTV Video Translation using Multi-task Adversarial Learning
There are large amount of valuable video archives in Video Home System (VHS) format. However, due to the analog nature, their quality is often poor. Compared to High-definition television (HDTV), VHS video not only has a…
Generative Adversarial NetworkSuper-ResolutionTranslationUnsupervised Adversarial Visual Level Domain Adaptation for Learning Video Object Detectors from Images
Deep learning based object detectors require thousands of diversified bounding box and class annotated examples. Though image object detectors have shown rapid progress in recent years with the release of multiple large-…
Domain AdaptationImage-to-Image TranslationObjectobject-detection+3