Assessing The Potential Of Mid-Sized Language Models For Clinical QA
Large language models, such as GPT-4 and Med-PaLM, have shown impressive performance on clinical tasks; however, they require access to compute, are closed-source, and cannot be deployed on device. Mid-size models such as BioGPT-large, BioMedLM, LLaMA 2, and Mistral 7B avoid these drawbacks, but their capacity for clinical tasks has been understudied. To help assess their potential for clinical use and help researchers decide which model they should use, we compare their performance on two clinical question-answering (QA) tasks: MedQA and consumer query answering. We find that Mistral 7B is the best performing model, winning on all benchmarks and outperforming models trained specifically for the biomedical domain. While Mistral 7B's MedQA score of 63.0% approaches the original Med-PaLM, and it often can produce plausible responses to consumer health queries, room for improvement still exists. This study provides the first head-to-head assessment of open source mid-sized models on clinical tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
MedQAQuestion AnsweringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Rephrasing Electronic Health Records for Pretraining Clinical Language Models
Clinical language models are important for many applications in healthcare, but their development depends on access to extensive clinical text for pretraining. However, obtaining clinical notes from electronic health rec…
DecoderLanguage ModelingLanguage ModellingFrom RAGs to riches: Using large language models to write documents for clinical trials
Clinical trials require numerous documents to be written -- protocols, consent forms, clinical study reports and others. Large language models (LLMs) offer the potential to rapidly generate first versions of these docume…
RAGRetrievalRetrieval-augmented GenerationAutomatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
Large Language Models (LLMs) have demonstrated remarkable proficiency in human interactions, yet their application within the medical field remains insufficiently explored. Previous works mainly focus on the performance …
Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement
Recently, vision-language representation learning has made remarkable advancements in building up medical foundation models, holding immense potential for transforming the landscape of clinical research and medical care.…
Medical Image AnalysisPhrase GroundingRepresentation LearningHarnessing the Power of Deep Learning Methods in Healthcare: Neonatal Pain Assessment from Crying Sound
Neonatal pain assessment in clinical environments is challenging as it is discontinuous and biased. Facial/body occlusion can occur in such settings due to clinical condition, developmental delays, prone position, or oth…
Position