Using Two Losses and Two Datasets Simultaneously to Improve TempoWiC Accuracy
WSD (Word Sense Disambiguation) is the task of identifying which sense of a word is meant in a sentence or other segment of text. Researchers have worked on this task (e.g. Pustejovsky, 2002) for years but it's still a challenging one even for SOTA (state-of-the-art) LMs (language models). The new dataset, TempoWiC introduced by Loureiro et al. (2022b) focuses on the fact that words change over time. Their best baseline achieves 70.33% macro-F1. In this work, we use two different losses simultaneously to train RoBERTa-based classification models. We also improve our model by using another similar dataset to generalize better. Our best configuration beats their best baseline by 4.23% and reaches 74.56% macroF1.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceVocal Bursts Valence PredictionWord Sense DisambiguationSimilar Papers 제목 키워드 기반
TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media
Language evolves over time, and word meaning changes accordingly. This is especially true in social media, since its dynamic nature leads to faster semantic shifts, making it challenging for NLP models to deal with new c…
Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC
This paper mainly describes the dma submission to the TempoWiC task, which achieves a macro-F1 score of 77.05% and attains the first place in this task. We first explore the impact of different pre-trained language model…
Data AugmentationMixture-of-ExpertsPOSRevisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Framework
As a seminal tool in self-supervised representation learning, contrastive learning has gained unprecedented attention in recent years. In essence, contrastive learning aims to leverage pairs of positive and negative samp…
Contrastive LearningRepresentation LearningBD-KD: Balancing the Divergences for Online Knowledge Distillation
We address the challenge of producing trustworthy and accurate compact models for edge devices. While Knowledge Distillation (KD) has improved model compression in terms of achieving high accuracy performance, calibratio…
Knowledge DistillationModel CompressionTransfer LearningPrincipled Approaches for Learning to Defer with Multiple Experts
We present a study of surrogate losses and algorithms for the general problem of learning to defer with multiple experts. We first introduce a new family of surrogate losses specifically tailored for the multiple-expert …