Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of PRIMATE Dataset
This paper addresses the quality of annotations in mental health datasets used for NLP-based depression level estimation from social media texts. While previous research relies on social media-based datasets annotated with binary categories, i.e. depressed or non-depressed, recent datasets such as D2S and PRIMATE aim for nuanced annotations using PHQ-9 symptoms. However, most of these datasets rely on crowd workers without the domain knowledge for annotation. Focusing on the PRIMATE dataset, our study reveals concerns regarding annotation validity, particularly for the lack of interest or pleasure symptom. Through reannotation by a mental health professional, we introduce finer labels and textual spans as evidence, identifying a notable number of false positives. Our refined annotations, to be released under a Data Use Agreement, offer a higher-quality test set for anhedonia detection. This study underscores the necessity of addressing annotation quality issues in mental health datasets, advocating for improved methodologies to enhance NLP model reliability in mental health assessments.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Prediction of depression status in college students using a Naive Bayes classifier based machine learning model
This study presents a machine learning model based on the Naive Bayes classifier for predicting the level of depression in university students, the objective was to improve prediction accuracy using a machine learning mo…
SpecificitySemantic Similarity Models for Depression Severity Estimation
Depressive disorders constitute a severe public health issue worldwide. However, public health systems have limited capacity for case detection and diagnosis. In this regard, the widespread use of social media has opened…
Semantic SimilaritySemantic Textual SimilarityTag2Risk: Harnessing Social Music Tags for Characterizing Depression Risk
Musical preferences have been considered a mirror of the self. In this age of Big Data, online music streaming services allow us to capture ecologically valid music listening behavior and provide a rich source of informa…
validPrediction-Constrained Topic Models for Antidepressant Recommendation
Supervisory signals can help topic models discover low-dimensional data representations that are more interpretable for clinical tasks. We propose a framework for training supervised latent Dirichlet allocation that bala…
PredictionTopic ModelsEmotional Vietnamese Speech-Based Depression Diagnosis Using Dynamic Attention Mechanism
Major depressive disorder is a prevalent and serious mental health condition that negatively impacts your emotions, thoughts, actions, and overall perception of the world. It is complicated to determine whether a person …
Speech Emotion Recognition