RepBERT: Contextualized Text Embeddings for First-Stage Retrieval
Although exact term match between queries and documents is the dominant method to perform first-stage retrieval, we propose a different approach, called RepBERT, to represent documents and queries with fixed-length contextualized embeddings. The inner products of query and document embeddings are regarded as relevance scores. On MS MARCO Passage Ranking task, RepBERT achieves state-of-the-art results among all initial retrieval techniques. And its efficiency is comparable to bag-of-words methods.
Code (3)
Tasks
Passage RankingRetrievalSimilar Papers 제목 키워드 기반
Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings
Contextualized word embeddings in language models have given much advance to NLP. Intuitively, sentential information is integrated into the representation of words, which can help model polysemy. However, context sensit…
PositionSentenceWord EmbeddingsWord Sense DisambiguationWhat Does This Word Mean? Explaining Contextualized Embeddings with Natural Language Definition
Contextualized word embeddings have boosted many NLP tasks compared with traditional static word embeddings. However, the word with a specific sense may have different contextualized embeddings due to its various context…
Word EmbeddingsIncremental Sense Weight Training for the Interpretation of Contextualized Word Embeddings
We present a novel online algorithm that learns the essence of each dimension in word embeddings by minimizing the within-group distance of contextualized embedding groups. Three state-of-the-art neural-based language mo…
Word EmbeddingsWord Sense DisambiguationKeyphrase Extraction from Scholarly Articles as Sequence Labeling using Contextualized Embeddings
In this paper, we formulate keyphrase extraction from scholarly articles as a sequence labeling task solved using a BiLSTM-CRF, where the words in the input text are represented using deep contextualized embeddings. We e…
ArticlesKeyphrase ExtractionWord EmbeddingsDynamic Contextualized Word Embeddings
Static word embeddings that represent words by a single vector cannot capture the variability of word meaning in different linguistic and extralinguistic contexts. Building on prior work on contextualized and dynamic wor…
Language ModelingLanguage ModellingWord Embeddings