Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings
We introduce a novel suite of state-of-the-art bilingual text embedding models that are designed to support English and another target language. These models are capable of processing lengthy text inputs with up to 8192 tokens, making them highly versatile for a range of natural language processing tasks such as text retrieval, clustering, and semantic textual similarity (STS) calculations. By focusing on bilingual models and introducing a unique multi-task learning objective, we have significantly improved the model performance on STS tasks, which outperforms the capabilities of existing multilingual models in both target language understanding and cross-lingual evaluation tasks. Moreover, our bilingual models are more efficient, requiring fewer parameters and less memory due to their smaller vocabulary needs. Furthermore, we have expanded the Massive Text Embedding Benchmark (MTEB) to include benchmarks for German and Spanish embedding models. This integration aims to stimulate further research and advancement in text embedding technologies for these languages.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningMulti-Task LearningRetrievalSemantic Textual SimilaritySTSText RetrievalSimilar Papers 제목 키워드 기반
VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning
Recent studies have demonstrated the potential of cross-lingual transferability by training a unified Transformer encoder for multiple languages. In addition to involving the masked language model objective, existing cro…
Contrastive LearningLanguage ModelingLanguage ModellingSentenceBilBOWA: Fast Bilingual Distributed Representations without Word Alignments
We introduce BilBOWA (Bilingual Bag-of-Words without Alignments), a simple and computationally-efficient model for learning bilingual distributed representations of words which can scale to large monolingual datasets and…
Cross-Lingual Document ClassificationDocument ClassificationGeneral ClassificationSentence+1LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understanding
Recent advancements in Large Language Models (LLMs) have demonstrated sophisticated capabilities, including the ability to process and comprehend extended contexts. These emergent capabilities necessitate rigorous evalua…
Long-Context UnderstandingInformation ExtractionQuestion AnsweringTokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
Tokenization is a foundational step in the text process of Large Language Models (LLMs). Texts must be first tokenized into token IDs, which are then input to LLMs. Inefficient tokenization results in long token-ID seque…
Investigating Cross-Lingual Alignment Methods for Contextualized Embeddings with Token-Level Evaluation
In this paper, we present a thorough investigation on methods that align pre-trained contextualized embeddings into shared cross-lingual context-aware embedding space, providing strong reference benchmarks for future con…
RetrievalSentenceSentence RetrievalWord Similarity