Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text
Sequence-to-sequence automatic speech recognition (ASR) models require large quantities of data to attain high performance. For this reason, there has been a recent surge in interest for unsupervised and semi-supervised training in such models. This work builds upon recent results showing notable improvements in semi-supervised training using cycle-consistency and related techniques. Such techniques derive training procedures and losses able to leverage unpaired speech and/or text data by combining ASR with Text-to-Speech (TTS) models. In particular, this work proposes a new semi-supervised loss combining an end-to-end differentiable ASR$\rightarrow$TTS loss with TTS$\rightarrow$ASR loss. The method is able to leverage both unpaired speech and text data to outperform recently proposed related techniques in terms of \%WER. We provide extensive results analyzing the impact of data quantity and speech and text modalities and show consistent gains across WSJ and Librispeech corpora. Our code is provided in ESPnet to reproduce the experiments.
Code (0)
등록된 구현이 없습니다.
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Semi-Supervised Image Classificationspeech-recognitionSpeech Recognitiontext-to-speechText to SpeechMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text
Self-supervised pre-training has been successful in both text and speech processing. Speech and text offer different but complementary information. The question is whether we are able to perform a speech-text joint pre-t…
intent-classificationIntent ClassificationLanguage ModelingLanguage ModellingListening while Speaking and Visualizing: Improving ASR through Multimodal Chain
Previously, a machine speech chain, which is based on sequence-to-sequence deep learning, was proposed to mimic speech perception and production behavior. Such chains separately processed listening and speaking by automa…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Data AugmentationImage Captioning+7Multimodal Sequential Generative Models for Semi-Supervised Language Instruction Following
Agents that can follow language instructions are expected to be useful in a variety of situations such as navigation. However, training neural network-based agents requires numerous paired trajectories and languages. Thi…
Instruction FollowingLearning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network
ASR has been shown to achieve great performance recently. However, most of them rely on massive paired data, which is not feasible for low-resource languages worldwide. This paper investigates how to learn directly from …
Acoustic Unit DiscoveryGenerative Adversarial NetworkSequence-Level Unsupervised Training in Speech Recognition: A Theoretical Study
Unsupervised speech recognition is a task of training a speech recognition model with unpaired data. To determine when and how unsupervised speech recognition can succeed, and how classification error relates to candidat…
Speech Recognition