Unsupervised Generative Adversarial Alignment Representation for Sheet music, Audio and Lyrics
Sheet music, audio, and lyrics are three main modalities during writing a song. In this paper, we propose an unsupervised generative adversarial alignment representation (UGAAR) model to learn deep discriminative representations shared across three major musical modalities: sheet music, lyrics, and audio, where a deep neural network based architecture on three branches is jointly trained. In particular, the proposed model can transfer the strong relationship between audio and sheet music to audio-lyrics and sheet-lyrics pairs by learning the correlation in the latent shared subspace. We apply CCA components of audio and sheet music to establish new ground truth. The generative (G) model learns the correlation of two couples of transferred pairs to generate new audio-sheet pair for a fixed lyrics to challenge the discriminative (D) model. The discriminative model aims at distinguishing the input which is from the generative model or the ground truth. The two models simultaneously train in an adversarial way to enhance the ability of deep alignment representation learning. Our experimental results demonstrate the feasibility of our proposed UGAAR for alignment representation learning among sheet music, audio, and lyrics.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningSimilar Papers 제목 키워드 기반
Unsupervised Lead Sheet Generation via Semantic Compression
Lead sheets have become commonplace in generative music research, being used as an initial compressed representation for downstream tasks like multitrack music generation and automatic arrangement. Despite this, research…
Music CompressionMusic GenerationSemantic CompressionLead Sheet Generation and Arrangement by Conditional Generative Adversarial Network
Research on automatic music generation has seen great progress due to the development of deep neural networks. However, the generation of multi-instrument music of arbitrary genres still remains a challenge. Existing res…
Generative Adversarial NetworkMusic GenerationWhy be adversarial? Let's cooperate!: Cooperative Dataset Alignment via JSD Upper Bound
Unsupervised dataset alignment estimates a transformation that maps two or more source domains to a shared aligned domain given only the domain datasets. This task has many applications including generative modeling, uns…
Domain AdaptationUnsupervised Domain AdaptationIterative Alignment Flows
The unsupervised task of aligning two or more distributions in a shared latent space has many applications including fair representations, batch effect mitigation, and unsupervised domain adaptation. Existing flow-based …
Domain AdaptationUnsupervised Domain AdaptationATOGAN:Adaptive Training Objective Generative Adversarial Network for Cross-lingual Word Alignment in Non-Isomorphic Embedding Spaces
Cross-lingual word alignment is a task for word translation from monolingual word embedding spaces of two languages. Recent works are mostly based on supervised approaches, which need specific bilingual seed dictionaries…
Generative Adversarial NetworkTranslationWord AlignmentWord Translation