mcBERT: Momentum Contrastive Learning with BERT for Zero-Shot Slot Filling
Zero-shot slot filling has received considerable attention to cope with the problem of limited available data for the target domain. One of the important factors in zero-shot learning is to make the model learn generalized and reliable representations. For this purpose, we present mcBERT, which stands for momentum contrastive learning with BERT, to develop a robust zero-shot slot filling model. mcBERT uses BERT to initialize the two encoders, the query encoder and key encoder, and is trained by applying momentum contrastive learning. Our experimental results on the SNIPS benchmark show that mcBERT substantially outperforms the previous models, recording a new state-of-the-art. Besides, we also show that each component composing mcBERT contributes to the performance improvement.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive Learningslot-fillingSlot FillingZero-Shot LearningZero-shot Slot FillingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Non-Contrastive Vision-Language Learning with Predictive Embedding Alignment
Vision-language models have transformed multimodal representation learning, yet dominant contrastive approaches like CLIP require large batch sizes, careful negative sampling, and extensive hyperparameter tuning. We intr…
Representation LearningMomentum Contrastive Pre-training for Question Answering
Existing pre-training methods for extractive Question Answering (QA) generate cloze-like queries different from natural questions in syntax structure, which could overfit pre-trained models to simple keyword matching. In…
BenchmarkingContrastive LearningExtractive Question-AnsweringNatural Questions+1Multi-Level Contrastive Learning for Cross-Lingual Alignment
Cross-language pre-trained models such as multilingual BERT (mBERT) have achieved significant performance in various cross-lingual downstream NLP tasks. This paper proposes a multi-level contrastive learning (ML-CTL) fra…
Contrastive LearningCross-Lingual TransferSentenceBoosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources
In medical healthcare, obtaining detailed annotations is challenging, highlighting the need for robust Vision-Language Models (VLMs). Pretrained VLMs enable fine-tuning on small datasets or zero-shot inference, achieving…
Computational EfficiencyContrastive LearningLabel-aware Hard Negative Sampling Strategies with Momentum Contrastive Learning for Implicit Hate Speech Detection
Detecting implicit hate speech that is not directly hateful remains a challenge. Recent research has attempted to detect implicit hate speech by applying contrastive learning to pre-trained language models such as BERT a…
Contrastive LearningHate Speech Detection