Evaluating ChatGPT text-mining of clinical records for obesity monitoring
Background: Veterinary clinical narratives remain a largely untapped resource for addressing complex diseases. Here we compare the ability of a large language model (ChatGPT) and a previously developed regular expression (RegexT) to identify overweight body condition scores (BCS) in veterinary narratives. Methods: BCS values were extracted from 4,415 anonymised clinical narratives using either RegexT or by appending the narrative to a prompt sent to ChatGPT coercing the model to return the BCS information. Data were manually reviewed for comparison. Results: The precision of RegexT was higher (100%, 95% CI 94.81-100%) than the ChatGPT (89.3%; 95% CI82.75-93.64%). However, the recall of ChatGPT (100%. 95% CI 96.18-100%) was considerably higher than that of RegexT (72.6%, 95% CI 63.92-79.94%). Limitations: Subtle prompt engineering is needed to improve ChatGPT output. Conclusions: Large language models create diverse opportunities and, whilst complex, present an intuitive interface to information but require careful implementation to avoid unpredictable errors.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingLarge Language ModelPrompt EngineeringSimilar Papers 제목 키워드 기반
Utilizing ChatGPT to Enhance Clinical Trial Enrollment
Clinical trials are a critical component of evaluating the effectiveness of new medical interventions and driving advancements in medical research. Therefore, timely enrollment of patients is crucial to prevent delays or…
Language ModelingLanguage ModellingLarge Language ModelRetrievalDoes Synthetic Data Generation of LLMs Help Clinical Text Mining?
Recent advancements in large language models (LLMs) have led to the development of highly potent models like OpenAI's ChatGPT. These models have exhibited exceptional performance in a variety of tasks, such as question a…
Code Generationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+3ChatLog: Carefully Evaluating the Evolution of ChatGPT Across Time
ChatGPT has achieved great success and can be considered to have acquired an infrastructural status. There are abundant works for evaluating ChatGPT on benchmarks. However, existing benchmarks encounter two challenges: (…
Natural Language UnderstandingAnnotation of negation in the IULA Spanish Clinical Record Corpus
This paper presents the IULA Spanish Clinical Record Corpus, a corpus of 3,194 sentences extracted from anonymized clinical records and manually annotated with negation markers and their scope. The corpus was conceived a…
Medical DiagnosisNegationNegation DetectionTerm ExtractionMedical Data Augmentation via ChatGPT: A Case Study on Medication Identification and Medication Event Classification
The identification of key factors such as medications, diseases, and relationships within electronic health records and clinical notes has a wide range of applications in the clinical field. In the N2C2 2022 competitions…
Data Augmentation