Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?
Self-supervised learning (SSL) has recently allowed leveraging large datasets of unlabeled speech signals to reach impressive performance on speech tasks using only small amounts of annotated data. The high number of proposed approaches fostered the need and rise of extended benchmarks that evaluate their performance on a set of downstream tasks exploring various aspects of the speech signal. However, and while the number of considered tasks has been growing, most rely upon a single decoding architecture that maps the frozen SSL representations to the downstream labels. This work investigates the robustness of such benchmarking results to changes in the decoder architecture. Interestingly, it appears that varying the architecture of the downstream decoder leads to significant variations in the leaderboards of most tasks. Concerningly, our study reveals that benchmarking using limited decoders may cause a counterproductive increase in the sizes of the developed SSL models.
Code (1)
Tasks
BenchmarkingDecoderSelf-Supervised LearningSimilar Papers 제목 키워드 기반
MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets
In this paper, we provide a new perspective on self-supervised speech models from how the training targets are obtained. We generalize the targets extractor into Offline Targets Extractor (Off-TE) and Online Targets Extr…
Automatic Speech RecognitionMulti-Task LearningRepresentation LearningSelf-Supervised Learning+2Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Heads
Self-supervised learning (SSL) leverages large datasets of unlabeled speech to reach impressive performance with reduced amounts of annotated data. The high number of proposed approaches fostered the emergence of compreh…
BenchmarkingSelf-Supervised LearningSelf-Supervised Speech Representation Learning: A Review
Although supervised deep learning has revolutionized speech and audio processing, it has necessitated the building of specialist models for individual tasks and application scenarios. It is likewise difficult to apply th…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)BenchmarkingRepresentation Learning+3Selective HuBERT: Self-Supervised Pre-Training for Target Speaker in Clean and Mixture Speech
Self-supervised pre-trained speech models were shown effective for various downstream speech processing tasks. Since they are mainly pre-trained to map input speech to pseudo-labels, the resulting representations are onl…
Speech Analysis of Language Varieties in Italy
Italy exhibits rich linguistic diversity across its territory due to the distinct regional languages spoken in different areas. Recent advances in self-supervised learning provide new opportunities to analyze Italy's lin…
Contrastive LearningSelf-Supervised Learning