Optimizing Sentence Embedding with Pseudo-Labeling and Model Ensembles: A Hierarchical Framework for Enhanced NLP Tasks
Sentence embedding tasks are important in natural language processing (NLP), but improving their performance while keeping them reliable is still hard. This paper presents a framework that combines pseudo-label generation and model ensemble techniques to improve sentence embeddings. We use external data from SimpleWiki, Wikipedia, and BookCorpus to make sure the training data is consistent. The framework includes a hierarchical model with an encoding layer, refinement layer, and ensemble prediction layer, using ALBERT-xxlarge, RoBERTa-large, and DeBERTa-large models. Cross-attention layers combine external context, and data augmentation techniques like synonym replacement and back-translation increase data variety. Experimental results show large improvements in accuracy and F1-score compared to basic models, and studies confirm that cross-attention and data augmentation make a difference. This work presents an effective way to improve sentence embedding tasks and lays the groundwork for future NLP research.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationPseudo LabelSentenceSentence EmbeddingSentence-EmbeddingSentence EmbeddingsSimilar Papers 제목 키워드 기반
A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings
Contrastive learning has shown great potential in unsupervised sentence embedding tasks, e.g., SimCSE. However, We find that these existing solutions are heavily affected by superficial features like the length of senten…
Contrastive LearningSemantic Textual SimilaritySentenceSentence Embedding+3A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings
Contrastive learning has shown great potential in unsupervised sentence embedding tasks, e.g., SimCSE \citep{gao2021simcse}. However, these existing solutions are heavily affected by superficial features like the length …
Contrastive LearningSemantic Textual SimilaritySentenceSentence Embedding+3Semi-Supervised Regression with Heteroscedastic Pseudo-Labels
Pseudo-labeling is a commonly used paradigm in semi-supervised learning, yet its application to semi-supervised regression (SSR) remains relatively under-explored. Unlike classification, where pseudo-labels are discrete …
Learning functional sections in medical conversations: iterative pseudo-labeling and human-in-the-loop approach
Medical conversations between patients and medical professionals have implicit functional sections, such as "history taking", "summarization", "education", and "care plan." In this work, we are interested in learning to …
SentencePseudo-labeling with Keyword Refining for Few-Supervised Video Captioning
Video captioning generate a sentence that describes the video content. Existing methods always require a number of captions (\eg, 10 or 20) per video to train the model, which is quite costly. In this work, we explore th…
Video Captioning