Meta-Embeddings Based On Self-Attention
Creating meta-embeddings for better performance in language modelling has received attention lately, and methods based on concatenation or merely calculating the arithmetic mean of more than one separately trained embeddings to perform meta-embeddings have shown to be beneficial. In this paper, we devise a new meta-embedding model based on the self-attention mechanism, namely the Duo. With less than 0.4M parameters, the Duo mechanism achieves state-of-the-art accuracy in text classification tasks such as 20NG. Additionally, we propose a new meta-embedding sequece-to-sequence model for machine translation, which to the best of our knowledge, is the first machine translation model based on more than one word-embedding. Furthermore, it has turned out that our model outperform the Transformer not only in terms of achieving a better result, but also a faster convergence on recognized benchmarks, such as the WMT 2014 English-to-French translation task.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModellingMachine Translationtext-classificationText ClassificationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Multilingual Meta-Embeddings for Code-Switching Named Entity Recognition
In this paper, we propose Multilingual Meta-Embeddings (MME), an effective method to learn multilingual representations by leveraging monolingual pre-trained embeddings. MME learns to utilize information from these embed…
Language IdentificationMMEnamed-entity-recognitionNamed Entity Recognition+1Frustratingly Easy Meta-Embedding -- Computing Meta-Embeddings by Averaging Source Word Embeddings
Creating accurate meta-embeddings from pre-trained source embeddings has received attention lately. Methods based on global and locally-linear transformation and concatenation have shown to produce accurate meta-embeddin…
Word EmbeddingsUnsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models
A variety of contextualised language models have been proposed in the NLP community, which are trained on diverse corpora to produce numerous Neural Language Models (NLMs). However, different NLMs have reported different…
Semantic Textual SimilaritySentenceSentence EmbeddingSentence-EmbeddingFAME: Feature-Based Adversarial Meta-Embeddings for Robust Input Representations
Combining several embeddings typically improves performance in downstream tasks as different embeddings encode different information. It has been shown that even models using embeddings from transformers still benefit fr…
NERPOSPOS TaggingSentence+2Transductive Linear Probing: A Novel Framework for Few-Shot Node Classification
Few-shot node classification is tasked to provide accurate predictions for nodes from novel classes with only few representative labeled nodes. This problem has drawn tremendous attention for its projection to prevailing…
ClassificationContrastive LearningMeta-LearningNode Classification+2