Is Language Modeling Enough? Evaluating Effective Embedding Combinations
Universal embeddings, such as BERT or ELMo, are useful for a broad set of natural language processing tasks like text classification or sentiment analysis. Moreover, specialized embeddings also exist for tasks like topic modeling or named entity disambiguation. We study if we can complement these universal embeddings with specialized embeddings. We conduct an in-depth evaluation of nine well known natural language understanding tasks with SentEval. Also, we extend SentEval with two additional tasks to the medical domain. We present PubMedSection, a novel topic classification dataset focussed on the biomedical domain. Our comprehensive analysis covers 11 tasks and combinations of six embeddings. We report that combined embeddings outperform state of the art universal embeddings without any embedding fine-tuning. We observe that adding topic model based embeddings helps for most tasks and that differing pre-training tasks encode complementary features. Moreover, we present new state of the art results on the MPQA and SUBJ tasks in SentEval.
Code (0)
등록된 구현이 없습니다.
Tasks
Entity DisambiguationGeneral ClassificationLanguage ModelingLanguage ModellingNatural Language UnderstandingSentiment Analysistext-classificationText ClassificationTopic ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Expert Concept-Modeling Ground Truth Construction for Word Embeddings Evaluation in Concept-Focused Domains
We present a novel, domain expert-controlled, replicable procedure for the construction of concept-modeling ground truths with the aim of evaluating the application of word embeddings. In particular, our method is design…
Embeddings EvaluationPhilosophyWord EmbeddingsBreaking the Softmax Bottleneck: A High-Rank RNN Language Model
We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that…
Language ModelingLanguage ModellingVocal Bursts Intensity PredictionWord EmbeddingsNeural Random Projections for Language Modelling
Neural network-based language models deal with data sparsity problems by mapping the large discrete space of words into a smaller continuous space of real-valued vectors. By learning distributed vector representations fo…
Language ModellingModels In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens
Standard pretrained language models operate on sequences of subword tokens without direct access to the characters that compose each token’s string representation. We probe the embedding layer of pretrained language mode…
Language ModelingLanguage ModellingModels In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens
Standard pretrained language models operate on sequences of subword tokens without direct access to the characters that compose each token's string representation. We probe the embedding layer of pretrained language mode…
Language ModelingLanguage Modelling