CalBERT - Code-mixed Adaptive Language representations using BERT
A code-mixed language is a type of language that involves the combination of two or more language varieties in its script or speech. Analysis of code-text is difficult to tackle because the language present is not consistent and does not work with existing monolingual approaches. We propose a novel approach to improve performance in Transformers by introducing an additional step called "Siamese Pre-Training", which allows pre-trained monolingual Transformers to adapt language representations for code-mixed languages with a few examples of code-mixed data. The proposed architectures beat the state of the art F1-score on the Sentiment Analysis for Indian Languages (SAIL) dataset, with the highest possible improvement being 5.1 points, while also achieving the state-of-the-art accuracy on the IndicGLUE Product Reviews dataset by beating the benchmark by 0.4 points.
Code (1)
Tasks
Natural Language UnderstandingSentiment AnalysisMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CalBERT - Code-mixed Adaptive Language representations using BERT
A code-mixed language is a type of language that involves the combination of two or more language varieties in its script or speech. Code-mixed language has become increasingly prevalent in recent times, especially on so…
Sentiment AnalysisClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Clinical notes contain information about patients that goes beyond structured data like lab values and medications. However, clinical notes have been underused relative to structured data, because notes are high-dimensio…
Readmission PredictionImproving Drug Identification in Overdose Death Surveillance using Large Language Models
The rising rate of drug-related deaths in the United States, largely driven by fentanyl, requires timely and accurate surveillance. However, critical overdose data are often buried in free-text coroner reports, leading t…
MedicalBERT: enhancing biomedical natural language processing using pretrained BERT-based model
Recent advances in natural language processing (NLP) have been driven bypretrained language models like BERT, RoBERTa, T5, and GPT. Thesemodels excel at understanding complex texts, but biomedical literature, withits dom…
Relation ExtractionQuestion AnsweringTransfer LearningBERT-based Ranking for Biomedical Entity Normalization
Developing high-performance entity normalization algorithms that can alleviate the term variation problem is of great interest to the biomedical community. Although deep learning-based methods have been successfully appl…
Word Embeddings