Boosting the Performance of Transformer Architectures for Semantic Textual Similarity
Semantic textual similarity is the task of estimating the similarity between the meaning of two texts. In this paper, we fine-tune transformer architectures for semantic textual similarity on the Semantic Textual Similarity Benchmark by tuning the model partially and then end-to-end. We experiment with BERT, RoBERTa, and DeBERTaV3 cross-encoders by approaching the problem as a binary classification task or a regression task. We combine the outputs of the transformer models and use handmade features as inputs for boosting algorithms. Due to worse test set results coupled with improvements on the validation set, we experiment with different dataset splits to further investigate this occurrence. We also provide an error analysis, focused on the edges of the prediction range.
Code (0)
등록된 구현이 없습니다.
Tasks
Binary ClassificationSemantic Textual SimilarityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Tricks and Plug-ins for Gradient Boosting with Transformers
Transformer architectures dominate modern NLP but often demand heavy computational resources and intricate hyperparameter tuning. To mitigate these challenges, we propose a novel framework, BoostTransformer, that augment…
Text ClassificationBoosting CNN-based Handwriting Recognition Systems with Learnable Relaxation Labeling
The primary challenge for handwriting recognition systems lies in managing long-range contextual dependencies, an issue that traditional models often struggle with. To mitigate it, attention mechanisms have recently been…
Handwriting RecognitionContextualized word senses: from attention to compositionality
The neural architectures of language models are becoming increasingly complex, especially that of Transformers, based on the attention mechanism. Although their application to numerous natural language processing tasks h…
Hierarchical Attention Transformer Architecture For Syntactic Spell Correction
The attention mechanisms are playing a boosting role in advancements in sequence-to-sequence problems. Transformer architecture achieved new state of the art results in machine translation, and it's variants are since be…
DecoderMachine TranslationSentenceTranslationSub-token ViT Embedding via Stochastic Resonance Transformers
Vision Transformer (ViT) architectures represent images as collections of high-dimensional vectorized tokens, each corresponding to a rectangular non-overlapping patch. This representation trades spatial granularity for …
Depth EstimationDepth PredictionImage RetrievalObject Discovery+5