Better than BERT but Worse than Baseline

2021-05-12 · Boxiang Liu, Jiaji Huang, Xingyu Cai, Kenneth Church

This paper compares BERT-SQuAD and Ab3P on the Abbreviation Definition Identification (ADI) task. ADI inputs a text and outputs short forms (abbreviations/acronyms) and long forms (expansions). BERT with reranking improves over BERT without reranking but fails to reach the Ab3P rule-based baseline. What is BERT missing? Reranking introduces two new features: charmatch and freq. The first feature identifies opportunities to take advantage of character constraints in acronyms and the second feature identifies opportunities to take advantage of frequency constraints across documents.

📄 PDF Abstract BibTeX arXiv:2105.05915

Code (0)

등록된 구현이 없습니다.

Tasks

Reranking

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

WordPiece 설명 없음

Are All Languages Created Equal in Multilingual BERT?

2020-05-18 · WS 2020 7 · Shijie Wu, Mark Dredze

Multilingual BERT (mBERT) trained on 104 languages has shown surprisingly good cross-lingual performance on several NLP tasks, even without explicit cross-lingual signals. However, these evaluations have focused on cross…

All Cross-Lingual Transfer Dependency Parsing named-entity-recognition+3

ur-iw-hnt at GermEval 2021: An Ensembling Strategy with Multiple BERT Models

2021-10-05 · GermEval 2021 9 · Hoai Nam Tran, Udo Kruschwitz

This paper describes our approach (ur-iw-hnt) for the Shared Task of GermEval2021 to identify toxic, engaging, and fact-claiming comments. We submitted three runs using an ensembling strategy by majority (hard) voting wi…

QiaoNing at SemEval-2020 Task 4: Commonsense Validation and Explanation system based on ensemble of language model

2020-09-06 · SEMEVAL 2020 · Pai Liu

In this paper, we present language model system submitted to SemEval-2020 Task 4 competition: "Commonsense Validation and Explanation". We participate in two subtasks for subtask A: validation and subtask B: Explanation.…

Language Modeling Language Modelling Transfer Learning

Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability

2021-03-12 · Wei-Tsung Kao, Hung-Yi Lee

This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications. To verify pre-trained models' transferability, we …

Classification General Classification text-classification Text Classification

Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models’ Transferability

2021-11-01 · Findings (EMNLP) 2021 11 · Wei-Tsung Kao, Hung-Yi Lee

Classification text-classification Text Classification