LaDiff ULMFiT: A Layer Differentiated training approach for ULMFiT
In our paper, we present Deep Learning models with a layer differentiated training method which were used for the SHARED TASK@ CONSTRAINT 2021 sub-tasks COVID19 Fake News Detection in English and Hostile Post Detection in Hindi. We propose a Layer Differentiated training procedure for training a pre-trained ULMFiT arXiv:1801.06146 model. We used special tokens to annotate specific parts of the tweets to improve language understanding and gain insights on the model making the tweets more interpretable. The other two submissions included a modified RoBERTa model and a simple Random Forest Classifier. The proposed approach scored a precision and f1 score of 0.96728972 and 0.967324832 respectively for sub-task "COVID19 Fake News Detection in English". Also, Coarse-Grained Hostility f1 Score and Weighted FineGrained f1 score of 0.908648 and 0.533907 respectively for sub-task Hostile Post Detection in Hindi. The proposed approach ranked 61st out of 164 in the sub-task "COVID19 Fake News Detection in English and 18th out of 45 in the sub-task Hostile Post Detection in Hindi".
Code (1)
Tasks
Fake News DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Low Resource Text Classification with ULMFit and Backtranslation
In computer vision, virtually every state-of-the-art deep learning system is trained with data augmentation. In text classification, however, data augmentation is less widely practiced because it must be performed before…
ClassificationData AugmentationGeneral Classificationtext-classification+1Inferring the source of official texts: can SVM beat ULMFiT?
Official Gazettes are a rich source of relevant information to the public. Their careful examination may lead to the detection of frauds and irregularities that may prevent mismanagement of public funds. This paper prese…
Text ClassificationTransfer LearningLearning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System
A medical dialogue system is essential for healthcare service as providing primary clinical advice and diagnoses. It has been gradually adopted and practiced in medical organizations in the form of a conversational bot, …
Decision MakingKnowledge DistillationLanguage ModellingTransfer LearningThe merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews
We evaluated the effectiveness of using language models, that were pre-trained in one domain, as the basis for a classification model in another domain: Dutch book reviews. Pre-trained language models have opened up new …
ClassificationGeneral ClassificationLanguage ModelingLanguage Modelling+2Universal Language Model Fine-Tuning with Subword Tokenization for Polish
Universal Language Model for Fine-tuning [arXiv:1801.06146] (ULMFiT) is one of the first NLP methods for efficient inductive transfer learning. Unsupervised pretraining results in improvements on many NLP tasks for Engli…
Language ModelingLanguage ModellingTransfer Learning