Fine-Tuning Transformers for Identifying Self-Reporting Potential Cases and Symptoms of COVID-19 in Tweets

2021-04-12 · NAACL (SMM4H) 2021 6 · Max Fleming, Priyanka Dondeti, Caitlin N. Dreisbach, Adam Poliak

We describe our straight-forward approach for Tasks 5 and 6 of 2021 Social Media Mining for Health Applications (SMM4H) shared tasks. Our system is based on fine-tuning Distill- BERT on each task, as well as first fine-tuning the model on the other task. We explore how much fine-tuning is necessary for accurately classifying tweets as containing self-reported COVID-19 symptoms (Task 5) or whether a tweet related to COVID-19 is self-reporting, non-personal reporting, or a literature/news mention of the virus (Task 6).

📄 PDF Abstract BibTeX arXiv:2104.05501

Code (1)

mfleming99/SMM4H_2021 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Weight Decay 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

SINAI@SMM4H’22: Transformers for biomedical social media text mining in Spanish

2022-10-01 · SMM4H (COLING) 2022 10 · Mariia Chizhikova, Pilar López-Úbeda, Manuel C. Díaz-Galiano, L. Alfonso Ureña-López 외

This paper covers participation of the SINAI team in Tasks 5 and 10 of the Social Media Mining for Health (#SSM4H) workshop at COLING-2022. These tasks focus on leveraging Twitter posts written in Spanish for healthcare …

1024m at SMM4H 2024: Tasks 3, 5 & 6 -- Ensembles of Transformers and Large Language Models for Medical Text Classification

2024-10-21 · Ram Mohan Rao Kadiyala, M. V. P. Chandra Sekhara Rao

Social media is a great source of data for users reporting information and regarding their health and how various things have had an effect on them. This paper presents various approaches using Transformers and Large Lan…

Binary Classification text-classification Text Classification

Classification of Tweets Self-reporting Adverse Pregnancy Outcomes and Potential COVID-19 Cases Using RoBERTa Transformers

2021-06-01 · NAACL (SMM4H) 2021 6 · Lung-Hao Lee, Man-Chen Hung, Chien-Huan Lu, Chang-Hao Chen 외

This study describes our proposed model design for SMM4H 2021 shared tasks. We fine-tune the language model of RoBERTa transformers and their connecting classifier to complete the classification tasks of tweets for adver…

Language Modeling Language Modelling

EURO-5K: When Does Domain Pretraining Matter? Benchmarking Transformers for EU Reporting Obligation Extraction

2026-06-02 · Marios Koniaris, Vasileios Kotronis, Eugenia Giannini, Panayiotis Tsanakas arxiv

Extracting reporting obligations from EU legislation is critical for assessing and reducing regulatory reporting burden. However, distinguishing reporting requirements from structurally similar provisions requires specia…

Domain Adaptation

Improving Visual Prompt Tuning for Self-supervised Vision Transformers

2023-06-08 · Seungryong Yoo, Eunji Kim, Dahuin Jung, Jungbeom Lee 외

Visual Prompt Tuning (VPT) is an effective tuning method for adapting pretrained Vision Transformers (ViTs) to downstream tasks. It leverages extra learnable tokens, known as prompts, which steer the frozen pretrained Vi…

image-classification Image Classification Semantic Segmentation Visual Prompt Tuning