eMLM: A New Pre-training Objective for Emotion Related Tasks
BERT has been shown to be extremely effective on a wide variety of natural language processing tasks, including sentiment analysis and emotion detection. However, the proposed pretraining objectives of BERT do not induce any sentiment or emotion-specific biases into the model. In this paper, we present Emotion Masked Language Modelling, a variation of Masked Language Modelling aimed at improving the BERT language representation model for emotion detection and sentiment analysis tasks. Using the same pre-training corpora as the original model, Wikipedia and BookCorpus, our BERT variation manages to improve the downstream performance on 4 tasks from emotion detection and sentiment analysis by an average of 1.2{\%} F-1. Moreover, our approach shows an increased performance in our task-specific robustness tests.
Code (1)
Tasks
Language ModellingSentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
KnowSemLM: A Knowledge Infused Semantic Language Model
Story understanding requires developing expectations of what events come next in text. Prior knowledge {--} both statistical and declarative {--} is essential in guiding such expectations. While existing semantic languag…
Cloze TestLanguage ModelingLanguage ModellingmodelAssemLM: A Spatial Reasoning Multimodal Large Language Model for Robotic Assembly
Spatial reasoning is a fundamental capability for embodied intelligence, especially for fine-grained manipulation tasks such as robotic assembly. Recent methods based on vision-language models (VLMs) largely rely on coar…
Spatial ReasoningPoint CloudsTransparency-First Medical Language Models: Datasheets, Model Cards, and End-to-End Data Provenance for Clinical NLP
We introduce TeMLM, a set of transparency-first release artifacts for clinical language models. TeMLM unifies provenance, data transparency, modeling transparency, and governance into a single, machine-checkable release …
Multi-Label ClassificationChemical Language Model Linker: blending text and molecules with modular adapters
The development of large language models and multi-modal models has enabled the appealing idea of generating novel molecules from text descriptions. Generative modeling would shift the paradigm from relying on large-scal…
Language ModelingLanguage Modellingmolecular representationExploring Embodied Multimodal Large Models: Development, Datasets, and Future Directions
Embodied multimodal large models (EMLMs) have gained significant attention in recent years due to their potential to bridge the gap between perception, cognition, and action in complex, real-world environments. This comp…
Decision Making