Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER
Contextual word embeddings (e.g. GPT, BERT, ELMo, etc.) have demonstrated state-of-the-art performance on various NLP tasks. Recent work with the multilingual version of BERT has shown that the model performs very well in zero-shot and zero-resource cross-lingual settings, where only labeled English data is used to finetune the model. We improve upon multilingual BERT's zero-resource cross-lingual performance via adversarial learning. We report the magnitude of the improvement on the multilingual MLDoc text classification and CoNLL 2002/2003 named entity recognition tasks. Furthermore, we show that language-adversarial training encourages BERT to align the embeddings of English documents and their translations, which may be the cause of the observed performance gains.
Code (0)
등록된 구현이 없습니다.
Tasks
General Classificationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NERtext-classificationText ClassificationWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Detecting Urgency Status of Crisis Tweets: A Transfer Learning Approach for Low Resource Languages
We release an urgency dataset that consists of English tweets relating to natural crises, along with annotations of their corresponding urgency status. Additionally, we release evaluation datasets for two low-resource la…
Transfer LearningXLM-RA Comparison of Architectures and Pretraining Methods for Contextualized Multilingual Word Embeddings
The lack of annotated data in many languages is a well-known challenge within the field of multilingual natural language processing (NLP). Therefore, many recent studies focus on zero-shot transfer learning and joint tra…
Multilingual Word Embeddingsnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+7Cross-lingual Dependency Parsing with Unlabeled Auxiliary Languages
Cross-lingual transfer learning has become an important weapon to battle the unavailability of annotated resources for low-resource languages. One of the fundamental techniques to transfer across languages is learning \e…
Cross-Lingual TransferDependency ParsingTransfer LearningWord EmbeddingsCross-Lingual BERT Transformation for Zero-Shot Dependency Parsing
This paper investigates the problem of learning cross-lingual representations in a contextual space. We propose Cross-Lingual BERT Transformation (CLBT), a simple and efficient approach to generate cross-lingual contextu…
Cross-Lingual TransferDependency ParsingLanguage ModelingLanguage Modelling+2Cross-lingual alignments of ELMo contextual embeddings
Building machine learning prediction models for a specific NLP task requires sufficient training data, which can be difficult to obtain for less-resourced languages. Cross-lingual embeddings map word embeddings from a le…
Dependency Parsingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+3