Guided contrastive self-supervised pre-training for automatic speech recognition

2022-10-22 · Aparna Khare, Minhua Wu, Saurabhchand Bhati, Jasha Droppo, Roland Maas

Contrastive Predictive Coding (CPC) is a representation learning method that maximizes the mutual information between intermediate latent representations and the output of a given model. It can be used to effectively initialize the encoder of an Automatic Speech Recognition (ASR) model. We present a novel modification of CPC called Guided Contrastive Predictive Coding (GCPC). Our proposed method maximizes the mutual information between representations from a prior-knowledge model and the output of the model being pre-trained, allowing prior knowledge injection during pre-training. We validate our method on 3 ASR tasks: German, French and English. Our method outperforms CPC pre-training on all three datasets, reducing the Word Error Rate (WER) by 4.44%, 6.55% and 15.43% relative on the German, French and English (Librispeech) tasks respectively, compared to training from scratch, while CPC pre-training only brings 2.96%, 1.01% and 14.39% relative WER reduction respectively.

📄 PDF Abstract BibTeX arXiv:2210.12335

Code (0)

등록된 구현이 없습니다.

Tasks

Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Representation Learning speech-recognition Speech Recognition

Methods 이 논문이 사용한 방법론

InfoNCE 설명 없음

Contrastive Predictive Coding Contrastive Predictive Coding (CPC) learns self-supervised representations by predicting the future in latent space by using powerful autoregressive models. The model uses a…

Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision

2026-03-14 · Yunhe Gao, Yabin Zhang, Chong Wang, Jiaming Liu 외 arxiv

Foundation models have transformed vision and language by learning general-purpose representations from large-scale unlabeled data, yet 3D medical imaging lacks analogous approaches. Existing self-supervised methods rely…

Self-Supervised Learning

Self-Contrastive Learning based Semi-Supervised Radio Modulation Classification

2022-03-29 · Dongxin Liu, Peng Wang, Tianshi Wang, Tarek Abdelzaher

This paper presents a semi-supervised learning framework that is new in being designed for automatic modulation classification (AMC). By carefully utilizing unlabeled signal data with a self-supervised contrastive-learni…

Classification Contrastive Learning

Semi-Supervised Contrastive Learning of Musical Representations

2024-07-18 · Julien Guinot, Elio Quinton, György Fazekas

Despite the success of contrastive learning in Music Information Retrieval, the inherent ambiguity of contrastive self-supervision presents a challenge. Relying solely on augmentation chains and self-supervised positive …

Contrastive Learning Information Retrieval Music Information Retrieval Transfer Learning

Guided Point Contrastive Learning for Semi-supervised Point Cloud Semantic Segmentation

2021-10-15 · ICCV 2021 10 · Li Jiang, Shaoshuai Shi, Zhuotao Tian, Xin Lai 외

Rapid progress in 3D semantic segmentation is inseparable from the advances of deep network models, which highly rely on large-scale annotated data for training. To address the high cost and challenges of 3D point-level …

3D Semantic Segmentation Contrastive Learning Pseudo Label Semantic Segmentation

A Brief Summary of Interactions Between Meta-Learning and Self-Supervised Learning

2021-03-01 · Huimin Peng

This paper briefly reviews the connections between meta-learning and self-supervised learning. Meta-learning can be applied to improve model generalization capability and to construct general AI algorithms. Self-supervis…

Data Augmentation Meta-Learning Self-Supervised Learning Unsupervised Pre-training