Knowledge Enhanced Masked Language Model for Stance Detection
Detecting stance on Twitter is especially challenging because of the short length of each tweet, the continuous coinage of new terminology and hashtags, and the deviation of sentence structure from standard prose. Fine-tuned language models using large-scale in-domain data have been shown to be the new state-of-the-art for many NLP tasks, including stance detection. In this paper, we propose a novel BERT-based fine-tuning method that enhances the masked language model for stance detection. Instead of random token masking, we propose using a weighted log-odds-ratio to identify words with high stance distinguishability and then model an attention mechanism that focuses on these words. We show that our proposed approach outperforms the state of the art for stance detection on Twitter data about the 2020 US Presidential election.
Code (1)
Tasks
Language ModelingLanguage ModellingSentenceStance DetectionStance Detection (US Election 2020 - Biden)Stance Detection (US Election 2020 - Trump)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection
Much of natural language processing is focused on leveraging large capacity language models, typically trained over single messages with a task of predicting one or more tokens. However, modeling human language at higher…
AttributeLanguage ModelingLanguage ModellingMasked Language Modeling+1Relative Counterfactual Contrastive Learning for Mitigating Pretrained Stance Bias in Stance Detection
Stance detection classifies stance relations (namely, Favor, Against, or Neither) between comments and targets. Pretrained language models (PLMs) are widely used to mine the stance relation to improve the performance of …
Contrastive LearningcounterfactualLanguage ModelingLanguage Modelling+2Knowledge-enhanced Prompt-tuning for Stance Detection
Investigating public attitudes on social media is important in opinion mining systems. Stance detection aims to analyze the attitude of an opinionated text (e.g., favor, neutral, or against) toward a given target. Ex …
Opinion MiningStance DetectionHow Vision Affects Language: Comparing Masked Self-Attention in Uni-Modal and Multi-Modal Transformer
The problem of interpretation of knowledge learned by multi-head self-attention in transformers has been one of the central questions in NLP. However, a lot of work mainly focused on models trained for uni-modal tasks, e…
Image CaptioningMachine TranslationText GenerationTranslationEnhanced Masked Image Modeling for Analysis of Dental Panoramic Radiographs
The computer-assisted radiologic informative report has received increasing research attention to facilitate diagnosis and treatment planning for dental care providers. However, manual interpretation of dental images is …
Instance SegmentationSelf-Supervised LearningSemantic Segmentation