Dynamic Meta-Embeddings for Improved Sentence Representations
While one of the first steps in many NLP systems is selecting what pre-trained word embeddings to use, we argue that such a step is better left for neural networks to figure out by themselves. To that end, we introduce dynamic meta-embeddings, a simple yet effective method for the supervised learning of embedding ensembles, which leads to state-of-the-art performance within the same model class on a variety of tasks. We subsequently show how the technique can be used to shed new light on the usage of word embeddings in NLP systems.
Code (3)
Tasks
SentenceWord EmbeddingsSimilar Papers 제목 키워드 기반
Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models
A variety of contextualised language models have been proposed in the NLP community, which are trained on diverse corpora to produce numerous Neural Language Models (NLMs). However, different NLMs have reported different…
Semantic Textual SimilaritySentenceSentence EmbeddingSentence-EmbeddingZero-Shot Audio Classification via Semantic Embeddings
In this paper, we study zero-shot learning in audio classification via semantic embeddings extracted from textual labels and sentence descriptions of sound classes. Our goal is to obtain a classifier that is capable of r…
Audio ClassificationClassificationGeneral ClassificationSentence+3LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States
Sentence representations are foundational to many Natural Language Processing (NLP) applications. While recent methods leverage Large Language Models (LLMs) to derive sentence representations, most rely on final-layer hi…
Sentence Meta-Embeddings for Unsupervised Semantic Textual Similarity
We address the task of unsupervised Semantic Textual Similarity (STS) by ensembling diverse pre-trained sentence encoders into sentence meta-embeddings. We apply, extend and evaluate different meta-embedding methods from…
Dimensionality ReductionSemantic Textual SimilaritySentenceSTS+1FAME: Feature-Based Adversarial Meta-Embeddings for Robust Input Representations
Combining several embeddings typically improves performance in downstream tasks as different embeddings encode different information. It has been shown that even models using embeddings from transformers still benefit fr…
NERPOSPOS TaggingSentence+2