Assessing the Impact of Anisotropy in Neural Representations of Speech: A Case Study on Keyword Spotting
Pretrained speech representations like wav2vec2 and HuBERT exhibit strong anisotropy, leading to high similarity between random embeddings. While widely observed, the impact of this property on downstream tasks remains unclear. This work evaluates anisotropy in keyword spotting for computational documentary linguistics. Using Dynamic Time Warping, we show that despite anisotropy, wav2vec2 similarity measures effectively identify words without transcription. Our results highlight the robustness of these representations, which capture phonetic structures and generalize across speakers. Our results underscore the importance of pretraining in learning rich and invariant speech representations.
Code (0)
등록된 구현이 없습니다.
Tasks
Dynamic Time WarpingKeyword SpottingSimilar Papers 제목 키워드 기반
Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing
Language models strongly rely on frequency information because they maximize the likelihood of tokens during pre-training. As a consequence, language models tend to not generalize well to tokens that are seldom seen duri…
Language ModelingLanguage ModellingSentenceIs there Anisotropy in Structural Bias?
Structural Bias (SB) is an important type of algorithmic deficiency within iterative optimisation heuristics. However, methods for detecting structural bias have not yet fully matured, and recent studies have uncovered m…
Towards objective and interpretable speech disorder assessment: a comparative analysis of CNN and transformer-based models
Head and Neck Cancers (HNC) significantly impact patients' ability to speak, affecting their quality of life. Commonly used metrics for assessing pathological speech are subjective, prompting the need for automated and u…
Anisotropy Is Inherent to Self-Attention in Transformers
The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotropy, a singular property of hidden represe…
Self-Supervised LearningIs Anisotropy Inherent to Transformers?
The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotropy, a singular property of hidden represe…
Self-Supervised Learning