Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC
This paper mainly describes the dma submission to the TempoWiC task, which achieves a macro-F1 score of 77.05% and attains the first place in this task. We first explore the impact of different pre-trained language models. Then we adopt data cleaning, data augmentation, and adversarial training strategies to enhance the model generalization and robustness. For further improvement, we integrate POS information and word semantic representation using a Mixture-of-Experts (MoE) approach. The experimental results show that MoE can overcome the feature overuse issue and combine the context, POS, and word semantic features well. Additionally, we use a model ensemble method for the final prediction, which has been proven effective by many research works.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationMixture-of-ExpertsPOSMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Methodology for Building a Diachronic Dataset of Semantic Shifts and its Application to QC-FR-Diac-V1.0, a Free Reference for French
Different algorithms have been proposed to detect semantic shifts (changes in a word meaning over time) in a diachronic corpus. Yet, and somehow surprisingly, no reference corpus has been designed so far to evaluate them…
Semantic Shift DetectionWord Sense DisambiguationWhat is Done is Done: an Incremental Approach to Semantic Shift Detection
Contextual word embedding techniques for semantic shift detection are receiving more and more attention. In this paper, we present What is Done is Done (WiDiD), an incremental approach to semantic shift detection based o…
ClusteringSemantic Shift DetectionWords are Malleable: Computing Semantic Shifts in Political and Media Discourse
Recently, researchers started to pay attention to the detection of temporal shifts in the meaning of words. However, most (if not all) of these approaches restricted their efforts to uncovering change over time, thus neg…
SEKE: Specialised Experts for Keyword Extraction
Keyword extraction involves identifying the most descriptive words in a document, allowing automatic categorisation and summarisation of large quantities of diverse textual data. Relying on the insight that real-world ke…
DescriptiveKeyword ExtractionMixture-of-ExpertsToward Effective Multi-Domain Rumor Detection in Social Networks Using Domain-Gated Mixture-of-Experts
Social media platforms have become key channels for spreading and tracking rumors due to their widespread accessibility and ease of information sharing. Rumors can continuously emerge across diverse domains and topics, o…