On the Use of Semantically-Aligned Speech Representations for Spoken Language Understanding
In this paper we examine the use of semantically-aligned speech representations for end-to-end spoken language understanding (SLU). We employ the recently-introduced SAMU-XLSR model, which is designed to generate a single embedding that captures the semantics at the utterance level, semantically aligned across different languages. This model combines the acoustic frame-level speech representation learning model (XLS-R) with the Language Agnostic BERT Sentence Embedding (LaBSE) model. We show that the use of the SAMU-XLSR model instead of the initial XLS-R model improves significantly the performance in the framework of end-to-end SLU. Finally, we present the benefits of using this model towards language portability in SLU.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningSentenceSentence EmbeddingSentence-EmbeddingSpeech Representation LearningSpoken Language UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Towards Comprehensive Semantic Speech Embeddings for Chinese Dialects
Despite having hundreds of millions of speakers, Chinese dialects lag behind Mandarin in speech and language technologies. Most varieties are primarily spoken, making dialect-to-Mandarin speech-LLMs (large language model…
Speech RecognitionPRISM: Prosody-Integrated Multi-Agent Reasoning Framework for Empathetic Spoken Dialogue
Empathetic spoken dialogue systems require not only semantically appropriate responses but also emotionally aligned prosodic expression. However, cascade pipelines often discard acoustic cues during speech-to-text conver…
Dialogue GenerationResponse GenerationSpeech SynthesisCoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving
Code-switching is a widely prevalent linguistic phenomenon in multilingual societies like India. Building speech-to-text models for code-switched speech is challenging due to limited availability of datasets. In this wor…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translationspeech-recognition+3Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text
Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and l…
TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling
Recent efforts target spoken language models (SLMs) that not only listen but also speak for more natural human-LLM interaction. Joint speech-text modeling is a promising direction to achieve this. However, the effectiven…
Language ModelingLanguage Modellingparameter-efficient fine-tuningSpeech Tokenization