HUBERT Untangles BERT to Improve Transfer across NLP Tasks

2019-10-25 · Mehrad Moradshahi, Hamid Palangi, Monica S. Lam, Paul Smolensky, Jianfeng Gao

We introduce HUBERT which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model. We show that there is shared structure between different NLP datasets that HUBERT, but not BERT, is able to learn and leverage. We validate the effectiveness of our model on the GLUE benchmark and HANS dataset. Our experiment results show that untangling data-specific semantics from general language structure is key for better transfer among NLP tasks.

📄 PDF Abstract BibTeX arXiv:1910.12647

Code (1)

Mehrad0711/HUBERT 공식 구현 pytorch

Tasks

Language ModelingLanguage Modelling

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Weight Decay 설명 없음

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

WordPiece 설명 없음

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction

2024-11-25 · Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, Karen Livescu 외

Sign language processing has traditionally relied on task-specific models,limiting the potential for transfer learning across tasks. We introduce SHuBERT (Sign Hidden-Unit BERT), a self-supervised transformer encoder tha…

Representation Learning Sign Language Recognition Sign Language Translation Transfer Learning

AfriHuBERT: A self-supervised speech representation model for African languages

2024-09-30 · Jesujoba O. Alabi, Xuechen Liu, Dietrich Klakow, Junichi Yamagishi

In this work, we present AfriHuBERT, an extension of mHuBERT-147, a compact self-supervised learning (SSL) model pretrained on 147 languages. While mHuBERT-147 covered 16 African languages, we expand this to 1,226 throug…

Automatic Speech Recognition Automatic Speech Recognition (ASR)Cross-corpus Language Identification+4

mHuBERT-147: A Compact Multilingual HuBERT Model

2024-06-10 · Marcely Zanon Boito, Vivek Iyer, Nikolaos Lagos, Laurent Besacier 외

We present mHuBERT-147, the first general-purpose massively multilingual HuBERT speech representation model trained on 90K hours of clean, open-license data. To scale up the multi-iteration HuBERT approach, we use faiss-…

Automatic Speech Recognition (ASR)Diversity model Speech Language Identification+1

HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition

2022-04-13 · Ji Won Yoon, Beom Jun Woo, Nam Soo Kim

Pre-training with self-supervised models, such as Hidden-unit BERT (HuBERT) and wav2vec 2.0, has brought significant improvements in automatic speech recognition (ASR). However, these models usually require an expensive …

Automatic Speech Recognition Automatic Speech Recognition (ASR)speech-recognition Speech Recognition

MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates

2026-03-24 · Zikang Huang, Meng Ge, Tianrui Wang, Xuanchen Li 외 arxiv

Self-supervised learning (SSL) has advanced speech processing. However, existing speech SSL methods typically assume a single sampling rate and struggle with mixed-rate data due to temporal resolution mismatch. To addres…

Self-Supervised Learning Speech Recognition