MEXMA: Token-level objectives improve sentence representations
Current pre-trained cross-lingual sentence encoders approaches use sentence-level objectives only. This can lead to loss of information, especially for tokens, which then degrades the sentence representation. We propose MEXMA, a novel approach that integrates both sentence-level and token-level objectives. The sentence representation in one language is used to predict masked tokens in another language, with both the sentence representation and all tokens directly updating the encoder. We show that adding token-level objectives greatly improves the sentence representation quality across several tasks. Our approach outperforms current pre-trained cross-lingual sentence encoders on bi-text mining as well as several downstream tasks. We also analyse the information encoded in our tokens, and how the sentence representation is built from them.
Code (1)
Tasks
SentenceSimilar Papers 제목 키워드 기반
Jointly Learning to Label Sentences and Tokens
Learning to construct text representations in end-to-end systems can be difficult, as natural languages are highly compositional and task-specific annotated datasets are often limited in size. Methods for directly superv…
Grammatical Error DetectionSentenceSentence ClassificationToken-level Adaptive Training for Neural Machine Translation
There exists a token imbalance phenomenon in natural language as different tokens appear with different frequencies, which leads to different learning difficulties for tokens in Neural Machine Translation (NMT). The vani…
DiversityMachine TranslationNMTTranslationFrustratingly Simple Pretraining Alternatives to Masked Language Modeling
Masked language modeling (MLM), a self-supervised pretraining objective, is widely used in natural language processing for learning text representations. MLM trains a model to predict a random sample of input tokens that…
Language ModelingLanguage ModellingMasked Language ModelingSentenceOn Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning
Incorporating contrastive learning objectives in sentence representation learning (SRL) has yielded significant improvements on many sentence-level NLP tasks. However, it is not well understood why contrastive learning w…
Contrastive LearningRepresentation LearningSentenceSentence SimilarityLong Text Generation by Modeling Sentence-Level and Discourse-Level Coherence
Generating long and coherent text is an important but challenging task, particularly for open-ended language generation tasks such as story generation. Despite the success in modeling intra-sentence coherence, existing g…
DecoderSemantic SimilaritySemantic Textual SimilaritySentence+2