Latent-space disentanglement with untrained generator networks for the isolation of different motion types in video data
Isolating different types of motion in video data is a highly relevant problem in video analysis. Applications can be found, for example, in dynamic medical or biological imaging, where the analysis and further processing of the dynamics of interest is often complicated by additional, unwanted dynamics, such as motion of the measurement subject. In this work, it is empirically shown that a representation of video data via untrained generator networks, together with a specific technique for latent space disentanglement that uses minimal, one-dimensional information on some of the underlying dynamics, allows to efficiently isolate different, highly non-linear motion types. In particular, such a representation allows to freeze any selection of motion types, and to obtain accurate independent representations of other dynamics of interest. Obtaining such a representation does not require any pre-training on a training data set, i.e., all parameters of the generator network are learned directly from a single video.
Code (1)
Tasks
DisentanglementImage ReconstructionSimilar Papers 제목 키워드 기반
Manifold Learning and Alignment with Generative Adversarial Networks
We present a generative adversarial network (GAN) that conducts manifold learning and alignment (MLA): A task to learn the multi-manifold structure underlying data and to align those manifolds without any correspondence …
DisentanglementGenerative Adversarial NetworkFEAT: Face Editing with Attention
Employing the latent space of pretrained generators has recently been shown to be an effective means for GAN-based face manipulation. The success of this approach heavily relies on the innate disentanglement of the laten…
DisentanglementYou Only Look Yourself: Unsupervised and Untrained Single Image Dehazing Neural Network
In this paper, we study two challenging and less-touched problems in single image dehazing, namely, how to make deep learning achieve image dehazing without training on the ground-truth clean image (unsupervised) and a i…
DisentanglementImage DehazingSingle Image DehazingDPE: Disentanglement of Pose and Expression for General Video Portrait Editing
One-shot video-driven talking face generation aims at producing a synthetic talking video by transferring the facial motion from a video to an arbitrary portrait image. Head pose and facial expression are always entangle…
DisentanglementFace GenerationTalking Face GenerationVideo EditingThe Hessian Penalty: A Weak Prior for Unsupervised Disentanglement
Existing disentanglement methods for deep generative models rely on hand-picked priors and complex encoder-based architectures. In this paper, we propose the Hessian Penalty, a simple regularization term that encourages …
Disentanglement