Subword ELMo

2019-09-18 · Jiangtong Li, Hai Zhao, Zuchao Li, Wei Bi, Xiaojiang Liu

Embedding from Language Models (ELMo) has shown to be effective for improving many natural language processing (NLP) tasks, and ELMo takes character information to compose word representation to train language models.However, the character is an insufficient and unnatural linguistic unit for word representation.Thus we introduce Embedding from Subword-aware Language Models (ESuLMo) which learns word representation from subwords using unsupervised segmentation over words.We show that ESuLMo can enhance four benchmark NLP tasks more effectively than ELMo, including syntactic dependency parsing, semantic role labeling, implicit discourse relation recognition and textual entailment, which brings a meaningful improvement over ELMo.

📄 PDF Abstract BibTeX arXiv:1909.08357

Code (0)

등록된 구현이 없습니다.

Tasks

Dependency Parsing Natural Language Inference Semantic Role Labeling

Methods 이 논문이 사용한 방법론

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

BiLSTM A Bidirectional LSTM, or biLSTM, is a sequence processing model that consists of two LSTMs: one taking the input in a forward direction, and the other in a backwards…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

ELMo Embeddings from Language Models, or ELMo, is a type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and…

Masked ELMo: An evolution of ELMo towards fully contextual RNN language models

2020-10-08 · Gregory Senay, Emmanuelle Salin

This paper presents Masked ELMo, a new RNN-based model for language model pre-training, evolved from the ELMo language model. Contrary to ELMo which only uses independent left-to-right and right-to-left contexts, Masked …

Language Modeling Language Modelling

CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters

2020-10-20 · COLING 2020 8 · Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji 외

Due to the compelling improvements brought by BERT, many recent representation models adopted the Transformer architecture as their main building block, consequently inheriting the wordpiece tokenization system despite i…

Clinical Concept Extraction Drug–drug Interaction Extraction Natural Language Inference Relation Extraction+1

ELMoLex: Connecting ELMo and Lexicon Features for Dependency Parsing

2018-10-01 · CONLL 2018 10 · Ganesh Jawahar, Benjamin Muller, Amal Fethi, Louis Martin 외

In this paper, we present the details of the neural dependency parser and the neural tagger submitted by our team {`}ParisNLP{'} to the CoNLL 2018 Shared Task on parsing from raw text to Universal Dependencies. We augmen…

Dependency Parsing Language Modeling Language Modelling

Gender Bias in Contextualized Word Embeddings

2019-04-05 · NAACL 2019 6 · Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell 외

In this paper, we quantify, analyze and mitigate gender bias exhibited in ELMo's contextualized word vectors. First, we conduct several intrinsic analyses and find that (1) training data for ELMo contains significantly m…

Word Embeddings

ELMo-NB at SemEval-2020 Task 7: Assessing Sense of Humor in EditedNews Headlines Using ELMo and NB

2020-12-01 · SEMEVAL 2020 · Enas Khwaileh, Muntaha A. Al-As{'}ad

Our approach is constructed to improve on a couple of aspects; preprocessing with an emphasis on humor sense detection, using embeddings from state-of-the-art language model(Elmo), and ensembling the results came up with…