BanglaEmbed: Efficient Sentence Embedding Models for a Low-Resource Language Using Cross-Lingual Distillation Techniques
Sentence-level embedding is essential for various tasks that require understanding natural language. Many studies have explored such embeddings for high-resource languages like English. However, low-resource languages like Bengali (a language spoken by almost two hundred and thirty million people) are still under-explored. This work introduces two lightweight sentence transformers for the Bangla language, leveraging a novel cross-lingual knowledge distillation approach. This method distills knowledge from a pre-trained, high-performing English sentence transformer. Proposed models are evaluated across multiple downstream tasks, including paraphrase detection, semantic textual similarity (STS), and Bangla hate speech detection. The new method consistently outperformed existing Bangla sentence transformers. Moreover, the lightweight architecture and shorter inference time make the models highly suitable for deployment in resource-constrained environments, making them valuable for practical NLP applications in low-resource languages.
Code (0)
등록된 구현이 없습니다.
Tasks
Hate Speech DetectionKnowledge DistillationSemantic Textual SimilaritySentenceSentence EmbeddingSentence-EmbeddingSTSMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
LuxEmbedder: A Cross-Lingual Approach to Enhanced Luxembourgish Sentence Embeddings
Sentence embedding models play a key role in various Natural Language Processing tasks, such as in Topic Modeling, Document Clustering and Recommendation Systems. However, these models rely heavily on parallel data, whic…
Recommendation SystemsSentenceSentence EmbeddingSentence-Embedding+1Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment
The field of cross-lingual sentence embeddings has recently experienced significant advancements, but research concerning low-resource languages has lagged due to the scarcity of parallel corpora. This paper shows that c…
RetrievalSentenceSentence EmbeddingSentence-Embedding+4Language-agnostic Representation from Multilingual Sentence Encoders for Cross-lingual Similarity Estimation
We propose a method to distill a language-agnostic meaning embedding from a multilingual sentence encoder. By removing language-specific information from the original embedding, we retrieve an embedding that fully repres…
Cross-Lingual Semantic Textual SimilarityMachine TranslationSemantic Textual SimilaritySentence+2Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining
Existing models of multilingual sentence embeddings require large parallel data resources which are not available for low-resource languages. We propose a novel unsupervised method to derive multilingual sentence embeddi…
Language ModelingLanguage ModellingMachine TranslationParallel Corpus Mining+4English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings
Universal cross-lingual sentence embeddings map semantically similar cross-lingual sentences into a shared embedding space. Aligning cross-lingual sentence embeddings usually requires supervised cross-lingual parallel se…
Contrastive LearningRetrievalSentenceSentence Embedding+3