Unsupervised Sentence Representations as Word Information Series: Revisiting TF--IDF
Sentence representation at the semantic level is a challenging task for Natural Language Processing and Artificial Intelligence. Despite the advances in word embeddings (i.e. word vector representations), capturing sentence meaning is an open question due to complexities of semantic interactions among words. In this paper, we present an embedding method, which is aimed at learning unsupervised sentence representations from unlabeled text. We propose an unsupervised method that models a sentence as a weighted series of word embeddings. The weights of the word embeddings are fitted by using Shannon's word entropies provided by the Term Frequency--Inverse Document Frequency (TF--IDF) transform. The hyperparameters of the model can be selected according to the properties of data (e.g. sentence length and textual gender). Hyperparameter selection involves word embedding methods and dimensionalities, as well as weighting schemata. Our method offers advantages over existing methods: identifiable modules, short-term training, online inference of (unseen) sentence representations, as well as independence from domain, external knowledge and language resources. Results showed that our model outperformed the state of the art in well-known Semantic Textual Similarity (STS) benchmarks. Moreover, our model reached state-of-the-art performance when compared to supervised and knowledge-based STS systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Open-Ended Question AnsweringSemantic Textual SimilaritySentenceSTSWord EmbeddingsSimilar Papers 제목 키워드 기반
Sentence Encoding with Tree-constrained Relation Networks
The meaning of a sentence is a function of the relations that hold between its words. We instantiate this relational view of semantics in a series of neural models based on variants of relation networks (RNs) which repre…
General ClassificationMachine TranslationRelationSentence+3Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
The recent tremendous success of unsupervised word embeddings in a multitude of applications raises the obvious question if similar methods could be derived to improve embeddings (i.e. semantic representations) of word s…
SentenceSentence EmbeddingsWord EmbeddingsMirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models
Recent work indicated that pretrained language models (PLMs) such as BERT and RoBERTa can be transformed into effective sentence and word encoders even via simple self-supervised techniques. Inspired by this line of work…
Contextualised Word RepresentationsContrastive LearningSentenceWord EmbeddingsEnhancing Unsupervised Sentence Similarity Methods with Deep Contextualised Word Representations
Calculating Semantic Textual Similarity (STS) plays a significant role in many applications such as question answering, document summarisation, information retrieval and information extraction. All modern state of the ar…
Contextualised Word RepresentationsInformation RetrievalQuestion AnsweringRetrieval+5Unsupervised Extractive Summarization with Heterogeneous Graph Embeddings for Chinese Document
In the scenario of unsupervised extractive summarization, learning high-quality sentence representations is essential to select salient sentences from the input document. Previous studies focus more on employing statisti…
Extractive SummarizationSentenceSentence EmbeddingsUnsupervised Extractive Summarization