Supervised Contextual Embeddings for Transfer Learning in Natural Language Processing Tasks
Pre-trained word embeddings are the primary method for transfer learning in several Natural Language Processing (NLP) tasks. Recent works have focused on using unsupervised techniques such as language modeling to obtain these embeddings. In contrast, this work focuses on extracting representations from multiple pre-trained supervised models, which enriches word embeddings with task and domain specific knowledge. Experiments performed in cross-task, cross-domain and cross-lingual settings indicate that such supervised embeddings are helpful, especially in the low-resource setting, but the extent of gains is dependent on the nature of the task and domain. We make our code publicly available.
Code (1)
Tasks
Language ModelingLanguage ModellingTransfer LearningWord EmbeddingsSimilar Papers 제목 키워드 기반
A Survey on Contextual Embeddings
Contextual embeddings, such as ELMo and BERT, move beyond global word representations like Word2Vec and achieve ground-breaking performance on a wide range of natural language processing tasks. Contextual embeddings assi…
Model CompressionSurveyEnhancing Unsupervised Sentence Similarity Methods with Deep Contextualised Word Representations
Calculating Semantic Textual Similarity (STS) plays a significant role in many applications such as question answering, document summarisation, information retrieval and information extraction. All modern state of the ar…
Contextualised Word RepresentationsInformation RetrievalQuestion AnsweringRetrieval+5Detecting Urgency Status of Crisis Tweets: A Transfer Learning Approach for Low Resource Languages
We release an urgency dataset that consists of English tweets relating to natural crises, along with annotations of their corresponding urgency status. Additionally, we release evaluation datasets for two low-resource la…
Transfer LearningXLM-RCross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing
We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meanin…
Cross-lingual zero-shot dependency parsingDependency ParsingFew-Shot LearningWord EmbeddingsMassive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP
Transfer learning, particularly approaches that combine multi-task learning with pre-trained contextualized embeddings and fine-tuning, have advanced the field of Natural Language Processing tremendously in recent years.…
Dependency ParsingLanguage ModelingLanguage ModellingMasked Language Modeling+5