Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces
Recent research has shown that word embedding spaces learned from text corpora of different languages can be aligned without any parallel data supervision. Inspired by the success in unsupervised cross-lingual word embeddings, in this paper we target learning a cross-modal alignment between the embedding spaces of speech and text learned from corpora of their respective modalities in an unsupervised fashion. The proposed framework learns the individual speech and text embedding spaces, and attempts to align the two spaces via adversarial training, followed by a refinement procedure. We show how our framework could be used to perform spoken word classification and translation, and the results on these two tasks demonstrate that the performance of our unsupervised alignment approach is comparable to its supervised counterpart. Our framework is especially useful for developing automatic speech recognition (ASR) and speech-to-text translation systems for low- or zero-resource languages, which have little parallel audio-text data for training modern supervised ASR and speech-to-text translation models, but account for the majority of the languages spoken across the world.
Code (0)
등록된 구현이 없습니다.
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Cross-Lingual Word Embeddingscross-modal alignmentspeech-recognitionSpeech RecognitionSpeech-to-TextSpeech-to-Text TranslationTranslationWord EmbeddingsSimilar Papers 제목 키워드 기반
Evaluation of Audio-Visual Alignments in Visually Grounded Speech Models
Systems that can find correspondences between multiple modalities, such as between speech and images, have great potential to solve different recognition and data analysis tasks in an unsupervised manner. This work studi…
Cross-Modal RetrievalObject LocalizationRetrievalSemantic RetrievalUnsupervised Mismatch Localization in Cross-Modal Sequential Data with Application to Mispronunciations Localization
Content mismatch usually occurs when data from one modality is translated to another, e.g. language learners producing mispronunciations (errors in speech) when reading a sentence (target text) aloud. However, most exist…
SentenceSLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
Unsupervised pre-training is now the predominant approach for both text and speech understanding. Self-attention models pre-trained on large amounts of unannotated data have been hugely successful when fine-tuned on down…
Language ModelingLanguage ModellingText MatchingText Normalization+2Soft Alignment of Modality Space for End-to-end Speech Translation
End-to-end Speech Translation (ST) aims to convert speech into target text within a unified model. The inherent differences between speech and text modalities often impede effective cross-modal and cross-lingual transfer…
Cross-Lingual TransferTranslationAdaptive Inner Speech-Text Alignment for LLM-based Speech Translation
Recent advancement of large language models (LLMs) has led to significant breakthroughs across various tasks, laying the foundation for the development of LLM-based speech translation systems. Existing methods primarily …
Cross-Modal RetrievalTranslation