Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA

2020-09-17 · EMNLP 2020 11 · Ieva Staliūnaitė, Ignacio Iacobacci

Many NLP tasks have benefited from transferring knowledge from contextualized word embeddings, however the picture of what type of knowledge is transferred is incomplete. This paper studies the types of linguistic phenomena accounted for by language models in the context of a Conversational Question Answering (CoQA) task. We identify the problematic areas for the finetuned RoBERTa, BERT and DistilBERT models through systematic error analysis - basic arithmetic (counting phrases), compositional semantics (negation and Semantic Role Labeling), and lexical semantics (surprisal and antonymy). When enhanced with the relevant linguistic knowledge through multitask learning, the models improve in performance. Ensembles of the enhanced models yield a boost between 2.2 and 2.7 points in F1 score overall, and up to 42.1 points in F1 on the hardest question classes. The results show differences in ability to represent compositional and lexical information between RoBERTa, BERT and DistilBERT.

📄 PDF Abstract BibTeX arXiv:2009.08257

Code (0)

등록된 구현이 없습니다.

Tasks

Conversational Question AnsweringNegationQuestion Answering Semantic Role LabelingWord Embeddings

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Weight Decay 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

RoBERTa 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Compressing Transformer-Based Semantic Parsing Models using Compositional Code Embeddings

2020-10-10 · Findings of the Association for Computational Linguistics 2020 · Prafull Prakash, Saurabh Kumar Shashidhar, Wenlong Zhao, Subendhu Rongali 외

The current state-of-the-art task-oriented semantic parsing models use BERT or RoBERTa as pretrained encoders; these models have huge memory footprints. This poses a challenge to their deployment for voice assistants suc…

Semantic Parsing

Felix&Julia at SemEval-2022 Task 4: Patronizing and Condescending Language Detection

2022-07-01 · SemEval (NAACL) 2022 7 · Felix Herrmann, Julia Krebs

This paper describes the authors’ submission to the SemEval-2022 task 4: Patronizing and Condescending Language (PCL) Detection. The aim of the task is the detection and classification of PCL in an annotated dataset. Sub…

Binary Classification Classification Multi-Label Classification MUlTI-LABEL-ClASSIFICATION

Exploring Transformers in Emotion Recognition: a comparison of BERT, DistillBERT, RoBERTa, XLNet and ELECTRA

2021-04-05 · Diogo Cortiz

This paper investigates how Natural Language Understanding (NLU) could be applied in Emotion Recognition, a specific task in affective computing. We finetuned different transformers language models (BERT, DistilBERT, RoB…

Emotion Recognition Natural Language Understanding

UnMASKed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts

2024-01-28 · Iñigo Parra

Language models (LMs) have become pivotal in the realm of technological advancements. While their capabilities are vast and transformative, they often include societal biases encoded in the human-produced datasets used f…

Do Language Models Learn Position-Role Mappings?

2022-02-08 · Jackson Petty, Michael Wilson, Robert Frank

How is knowledge of position-role mappings in natural language learned? We explore this question in a computational setting, testing whether a variety of well-performing pertained language models (BERT, RoBERTa, and Dist…

Position Specificity