Self-Supervised Speech Representations are More Phonetic than Semantic
Self-supervised speech models (S3Ms) have become an effective backbone for speech applications. Various analyses suggest that S3Ms encode linguistic properties. In this work, we seek a more fine-grained analysis of the word-level linguistic properties encoded in S3Ms. Specifically, we curate a novel dataset of near homophone (phonetically similar) and synonym (semantically similar) word pairs and measure the similarities between S3M word representation pairs. Our study reveals that S3M representations consistently and significantly exhibit more phonetic than semantic similarity. Further, we question whether widely used intent classification datasets such as Fluent Speech Commands and Snips Smartlights are adequate for measuring semantic abilities. Our simple baseline, using only the word identity, surpasses S3M-based models. This corroborates our findings and suggests that high scores on these datasets do not necessarily guarantee the presence of semantic content.
Code (1)
Tasks
intent-classificationIntent ClassificationSemantic SimilaritySemantic Textual SimilaritySimilar Papers 제목 키워드 기반
MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery
This paper introduces MauBERT, a multilingual extension of HuBERT that leverages articulatory features for robust cross-lingual phonetic representation learning. We continue HuBERT pre-training with supervision based on …
Self-Supervised LearningRepresentation LearningUniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both unlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastiv…
Multi-Task LearningRepresentation LearningSelf-Supervised Learningspeech-recognition+3An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speech
Self-supervised representation learning for speech often involves a quantization step that transforms the acoustic input into discrete units. However, it remains unclear how to characterize the relationship between these…
QuantizationRepresentation LearningProbing self-supervised speech models for phonetic and phonemic information: a case study in aspiration
Textless self-supervised speech models have grown in capabilities in recent years, but the nature of the linguistic information they encode has not yet been thoroughly examined. We evaluate the extent to which these mode…
Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Models
While discrete latent variable models have had great success in self-supervised learning, most models assume that frames are independent. Due to the segmental nature of phonemes in speech perception, modeling dependencie…
Self-Supervised Learning