GigaEmbeddings: Efficient Russian Language Embedding Model
We introduce GigaEmbeddings, a novel framework for training high-performance Russian-focused text embeddings through hierarchical instruction tuning of the decoder-only LLM designed specifically for Russian language (GigaChat-3B). Our three-stage pipeline, comprising large-scale contrastive pre-training in web-scale corpora, fine-tuning with hard negatives, and multitask generalization across retrieval, classification, and clustering tasks, addresses key limitations of existing methods by unifying diverse objectives and leveraging synthetic data generation. Architectural innovations include bidirectional attention for contextual modeling, latent attention pooling for robust sequence aggregation, and strategic pruning of 25% of transformer layers to enhance efficiency without compromising performance. Evaluated on the ruMTEB benchmark spanning 23 multilingual tasks, GigaEmbeddings achieves state-of-the-art results (69.1 avg. score), outperforming strong baselines with a larger number of parameters.
Code (0)
등록된 구현이 없습니다.
Tasks
Synthetic Data GenerationSimilar Papers 제목 키워드 기반
The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design
Embedding models play a crucial role in Natural Language Processing (NLP) by creating text embeddings used in various tasks such as information retrieval and assessing semantic text similarity. This paper focuses on rese…
Information RetrievalRerankingRetrievalSemantic Textual Similarity+3LANGUAGE MODEL EMBEDDINGS IMPROVE SENTIMENT ANALYSIS IN RUSSIAN
Sentiment analysis is one of the most popular natural language processing tasks. In this paper we introduce pre-trained Russian language models which are used to extract embeddings (ELMo) to improve accuracy for classifi…
ArticlesLanguage ModelingLanguage Modellingmodel+2Russian Language Datasets in the Digitial Humanities Domain and Their Evaluation with Word Embeddings
In this paper, we present Russian language datasets in the digital humanities domain for the evaluation of word embedding techniques or similar language modeling and feature learning algorithms. The datasets are split in…
Language ModelingLanguage ModellingWord EmbeddingsSentence Embeddings for Russian NLU
We investigate the performance of sentence embeddings models on several tasks for the Russian language. In our comparison, we include such tasks as multiple choice question answering, next sentence prediction, and paraph…
Multiple-choiceParaphrase IdentificationPredictionQuestion Answering+2Evaluation of Morphological Embeddings for the Russian Language
A number of morphology-based word embedding models were introduced in recent years. However, their evaluation was mostly limited to English, which is known to be a morphologically simple language. In this paper, we explo…
ChunkingNERPOSPOS Tagging+1