Chatbots Are Not Reliable Text Annotators
Recent research highlights the significant potential of ChatGPT for text annotation in social science research. However, ChatGPT is a closed-source product which has major drawbacks with regards to transparency, reproducibility, cost, and data protection. Recent advances in open-source (OS) large language models (LLMs) offer alternatives which remedy these challenges. This means that it is important to evaluate the performance of OS LLMs relative to ChatGPT and standard approaches to supervised machine learning classification. We conduct a systematic comparative evaluation of the performance of a range of OS LLM models alongside ChatGPT, using both zero- and few-shot learning as well as generic and custom prompts, with results compared to more traditional supervised classification models. Using a new dataset of Tweets from US news media, and focusing on simple binary text annotation tasks for standard social science concepts, we find significant variation in the performance of ChatGPT and OS models across the tasks, and that supervised classifiers consistently outperform both. Given the unreliable performance of ChatGPT and the significant challenges it poses to Open Science we advise against using ChatGPT for substantive text annotation tasks in social science research.
Code (1)
Tasks
Few-Shot Learningtext annotationSimilar Papers 제목 키워드 기반
On Overcoming Miscalibrated Conversational Priors in LLM-based Chatbots
We explore the use of Large Language Model (LLM-based) chatbots to power recommender systems. We observe that the chatbots respond poorly when they encounter under-specified requests (e.g., they make incorrect assumption…
Language ModelingLanguage ModellingLarge Language ModelRecommendation SystemsItem Response Theory for Efficient Human Evaluation of Chatbots
Conversational agent quality is currently assessed using human evaluation, and often requires an exorbitant number of comparisons to achieve statistical significance. In this paper, we introduce Item Response Theory (IRT…
ChatbotReliable Conversational Agents under ASP Control that Understand Natural Language
Efforts have been made to make machines converse like humans in the past few decades. The recent techniques of Large Language Models (LLMs) make it possible to have human-like conversations with machines, but LLM's flaws…
How Reliable AI Chatbots are for Disease Prediction from Patient Complaints?
Artificial Intelligence (AI) chatbots leveraging Large Language Models (LLMs) are gaining traction in healthcare for their potential to automate patient interactions and aid clinical decision-making. This study examines …
Decision MakingDisease PredictionFew-Shot LearningDid Chatbots Miss Their 'Apollo Moment'? A Survey of the Potential, Gaps and Lessons from Using Collaboration Assistants During COVID-19
Artificial Intelligence (AI) technologies have long been positioned as a tool to provide crucial data-driven decision support to people. In this survey paper, we look at how AI in general, and collaboration assistants (C…