How Emotionally Stable is ALBERT? Testing Robustness with Stochastic Weight Averaging on a Sentiment Analysis Task
Despite their success, modern language models are fragile. Even small changes in their training pipeline can lead to unexpected results. We study this phenomenon by examining the robustness of ALBERT (arXiv:1909.11942) in combination with Stochastic Weight Averaging (SWA) (arXiv:1803.05407) -- a cheap way of ensembling -- on a sentiment analysis task (SST-2). In particular, we analyze SWA's stability via CheckList criteria (arXiv:2005.04118), examining the agreement on errors made by models differing only in their random seed. We hypothesize that SWA is more stable because it ensembles model snapshots taken along the gradient descent trajectory. We quantify stability by comparing the models' mistakes with Fleiss' Kappa (Fleiss, 1971) and overlap ratio scores. We find that SWA reduces error rates in general; yet the models still suffer from their own distinct biases (according to CheckList).
Code (1)
Tasks
Sentiment AnalysisSST-2Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Phase transitions in the Prisoner's Dilemma game on the Barabási-Albert graph with participation cost
We examine the impact of the maintenance cost of social links on cooperative behavior in the Prisoner's Dilemma game on the Barab\'asi-Albert scale-free network with a pairwise stochastic imitation. We show by means of M…
Phase transitions in the Prisoner's Dilemma game on scale-free networks
We study stochastic dynamics of the Prisoner's Dilemma game on random Erd\"{o}s-R\'{e}nyi and Barab\'{a}si-Albert networks with a cost of maintaining a link between interacting players. Stochastic simulations show that w…
Decoding Linguistic Nuances in Mental Health Text Classification Using Expressive Narrative Stories
Recent advancements in NLP have spurred significant interest in analyzing social media text data for identifying linguistic features indicative of mental health issues. However, the domain of Expressive Narrative Stories…
text-classificationText ClassificationPhoneme-Level Deepfake Detection Across Emotional Conditions Using Self-Supervised Embeddings
Recent advances in emotional voice conversion (EVC) have enabled the generation of expressive synthetic speech, raising new concerns in audio deepfake detection. Existing approaches treat speech as a homogeneous signal a…
Audio Deepfake DetectionVoice ConversionImproving Drug Identification in Overdose Death Surveillance using Large Language Models
The rising rate of drug-related deaths in the United States, largely driven by fentanyl, requires timely and accurate surveillance. However, critical overdose data are often buried in free-text coroner reports, leading t…