KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms
In this paper, we propose a novel neural network model called KaraSinger for a less-studied singing voice synthesis (SVS) task named score-free SVS, in which the prosody and melody are spontaneously decided by machine. KaraSinger comprises a vector-quantized variational autoencoder (VQ-VAE) that compresses the Mel-spectrograms of singing audio to sequences of discrete codes, and a language model (LM) that learns to predict the discrete codes given the corresponding lyrics. For the VQ-VAE part, we employ a Connectionist Temporal Classification (CTC) loss to encourage the discrete codes to carry phoneme-related information. For the LM part, we use location-sensitive attention for learning a robust alignment between the input phoneme sequence and the output discrete code. We keep the architecture of both the VQ-VAE and LM light-weight for fast training and inference speed. We validate the effectiveness of the proposed design choices using a proprietary collection of 550 English pop songs sung by multiple amateur singers. The result of a listening test shows that KaraSinger achieves high scores in intelligibility, musicality, and the overall quality.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingSinging Voice SynthesisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Score and Lyrics-Free Singing Voice Generation
Generative models for singing voice have been mostly concerned with the task of ``singing voice synthesis,'' i.e., to produce singing voice waveforms given musical scores and text lyrics. In this work, we explore a novel…
Audio GenerationSinging Voice SynthesisMakeSinger: A Semi-Supervised Training Method for Data-Efficient Singing Voice Synthesis via Classifier-free Diffusion Guidance
In this paper, we propose MakeSinger, a semi-supervised training method for singing voice synthesis (SVS) via classifier-free diffusion guidance. The challenge in SVS lies in the costly process of gathering aligned sets …
Singing Voice Synthesistext-to-speechText to SpeechM4Singer: a Multi-Style, Multi-Singer and Musical Score Provided Mandarin Singing Corpus
The lack of publicly available high-quality and accurately labeled datasets has long been a major bottleneck for singing voice synthesis (SVS). To tackle this problem, we present M4Singer, a free-to-use Multi-style, Mult…
Music TranscriptionSinging Voice SynthesisVoice ConversionSinging voice synthesis based on convolutional neural networks
The present paper describes a singing voice synthesis based on convolutional neural networks (CNNs). Singing voice synthesis systems based on deep neural networks (DNNs) are currently being proposed and are improving the…
Singing Voice SynthesisFast and High-Quality Singing Voice Synthesis System based on Convolutional Neural Networks
The present paper describes singing voice synthesis based on convolutional neural networks (CNNs). Singing voice synthesis systems based on deep neural networks (DNNs) are currently being proposed and are improving the n…
Singing Voice Synthesis