Exploring the Feasibility of ChatGPT for Event Extraction
Event extraction is a fundamental task in natural language processing that involves identifying and extracting information about events mentioned in text. However, it is a challenging task due to the lack of annotated data, which is expensive and time-consuming to obtain. The emergence of large language models (LLMs) such as ChatGPT provides an opportunity to solve language tasks with simple prompts without the need for task-specific datasets and fine-tuning. While ChatGPT has demonstrated impressive results in tasks like machine translation, text summarization, and question answering, it presents challenges when used for complex tasks like event extraction. Unlike other tasks, event extraction requires the model to be provided with a complex set of instructions defining all event types and their schemas. To explore the feasibility of ChatGPT for event extraction and the challenges it poses, we conducted a series of experiments. Our results show that ChatGPT has, on average, only 51.04% of the performance of a task-specific model such as EEQA in long-tail and complex scenarios. Our usability testing experiments indicate that ChatGPT is not robust enough, and continuous refinement of the prompt does not lead to stable performance improvements, which can result in a poor user experience. Besides, ChatGPT is highly sensitive to different prompt styles.
Code (0)
등록된 구현이 없습니다.
Tasks
Event ExtractionMachine TranslationQuestion AnsweringText SummarizationSimilar Papers 제목 키워드 기반
Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study
With the advent of large language models (LLMs), there has been growing interest in exploring their potential for medical applications. This research aims to investigate the ability of LLMs, specifically ChatGPT, in the …
Data AugmentationEvent ExtractionPharmacovigilanceZero-shot Temporal Relation Extraction with ChatGPT
The goal of temporal relation extraction is to infer the temporal relation between two events in the document. Supervised models are dominant in this task. In this work, we investigate ChatGPT's ability on zero-shot temp…
RelationRelation ExtractionTemporal Relation ExtractionModel Leeching: An Extraction Attack Targeting LLMs
Model Leeching is a novel extraction attack targeting Large Language Models (LLMs), capable of distilling task-specific knowledge from a target LLM into a reduced parameter model. We demonstrate the effectiveness of our …
Adversarial AttackmodelEnhancing Complex Causality Extraction via Improved Subtask Interaction and Knowledge Fusion
Event Causality Extraction (ECE) aims at extracting causal event pairs from texts. Despite ChatGPT's recent success, fine-tuning small models remains the best approach for the ECE task. However, existing fine-tuning base…
In-Context LearningKnowledge GraphsLLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and Evaluation
Empowering chatbots in the field of mental health is receiving increasing amount of attention, while there still lacks exploration in developing and evaluating chatbots in psychiatric outpatient scenarios. In this work, …
ChatbotDiagnostic