Can tweets predict article retractions? A comparison between human and LLM labelling
Quickly detecting problematic research articles is crucial to safeguarding the integrity of scientific research. This study explores whether Twitter mentions of retracted articles can signal potential problems with the articles prior to their retraction, potentially serving as an early warning system for scholars. To investigate this, we analysed a dataset of 4,354 Twitter mentions associated with 504 retracted articles. The effectiveness of Twitter mentions in predicting article retractions was evaluated by both manual and Large Language Model (LLM) labelling. Manual labelling results indicated that 25.7% of tweets signalled problems before retraction. Using the manual labelling results as the baseline, we found that LLMs (GPT-4o-mini, Gemini 1.5 Flash, and Claude-3.5-Haiku) outperformed lexicon-based sentiment analysis tools (e.g., TextBlob) in detecting potential problems, suggesting that automatic detection of problematic articles from social media using LLMs is technically feasible. Nevertheless, since only a small proportion of retracted articles (11.1%) were criticised on Twitter prior to retraction, such automatic systems would detect only a minority of problematic articles. Overall, this study offers insights into how social media data, coupled with emerging generative AI techniques, can support research integrity.
Code (0)
등록된 구현이 없습니다.
Tasks
ArticlesLanguage ModelingLanguage ModellingLarge Language ModelSentiment AnalysisSimilar Papers 제목 키워드 기반
Linking Tweets with Monolingual and Cross-Lingual News using Transformed Word Embeddings
Social media platforms have grown into an important medium to spread information about an event published by the traditional media, such as news articles. Grouping such diverse sources of information that discuss the sam…
ArticlesWord EmbeddingsKnowBias: A Novel AI Method to Detect Polarity in Online Content
We propose a novel training and inference method for detecting political bias in long text content such as newspaper opinion articles. Obtaining long text data and annotations at sufficient scale for training is difficul…
ArticlesDomain AdaptationGeneral ClassificationSentence+2KnowBias: Detecting Political Polarity in Long Text Content
We introduce a classification scheme for detecting political bias in long text content such as newspaper opinion articles. Obtaining long text data and annotations at sufficient scale for training is difficult, but it is…
ArticlesDomain AdaptationGeneral ClassificationSentenceDiscriminating between standard Romanian and Moldavian tweets using filtered character ngrams
We applied word unigram models, character ngram models, and CNNs to the task of distinguishing tweets of two related dialects of Romanian (standard Romanian and Moldavian) for the VarDial 2020 RDI shared task (Gaman et a…
Articlestext-classificationText ClassificationRetractions: Updating from Complex Information
We modify a canonical experimental design to identify the effectiveness of retractions. Comparing beliefs after retractions to beliefs (a) without the retracted information and (b) after equivalent new information, we fi…
Experimental DesignMisinformation