Multimodal Large Language Models for Medicine: A Comprehensive Survey
MLLMs have recently become a focal point in the field of artificial intelligence research. Building on the strong capabilities of LLMs, MLLMs are adept at addressing complex multi-modal tasks. With the release of GPT-4, MLLMs have gained substantial attention from different domains. Researchers have begun to explore the potential of MLLMs in the medical and healthcare domain. In this paper, we first introduce the background and fundamental concepts related to LLMs and MLLMs, while emphasizing the working principles of MLLMs. Subsequently, we summarize three main directions of application within healthcare: medical reporting, medical diagnosis, and medical treatment. Our findings are based on a comprehensive review of 330 recent papers in this area. We illustrate the remarkable capabilities of MLLMs in these domains by providing specific examples. For data, we present six mainstream modes of data along with their corresponding evaluation benchmarks. At the end of the survey, we discuss the challenges faced by MLLMs in the medical and healthcare domain and propose feasible methods to mitigate or overcome these issues.
Code (0)
등록된 구현이 없습니다.
Tasks
Medical DiagnosisSurveyMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
Since the release of ChatGPT and GPT-4, large language models (LLMs) and multimodal large language models (MLLMs) have attracted widespread attention for their exceptional capabilities in understanding, reasoning, and ge…
SurveyLarge Language Models in Bioinformatics: A Survey
Large Language Models (LLMs) are revolutionizing bioinformatics, enabling advanced analysis of DNA, RNA, proteins, and single-cell data. This survey provides a systematic review of recent advancements, focusing on genomi…
SurveyA Survey of LLM-based Agents in Medicine: How far are we from Baymax?
Large Language Models (LLMs) are transforming healthcare through the development of LLM-based agents that can understand, reason about, and assist with medical tasks. This survey provides a comprehensive review of LLM-ba…
HallucinationSurveyA Comprehensive Survey of Foundation Models in Medicine
Foundation models (FMs) are large-scale deep learning models trained on massive datasets, often using self-supervised learning techniques. These models serve as a versatile base for a wide range of downstream tasks, incl…
Graph LearningMedical Image AnalysisSelf-Supervised LearningSurveyLarge Language Model Benchmarks in Medical Tasks
With the increasing application of large language models (LLMs) in the medical domain, evaluating these models' performance using benchmark datasets has become crucial. This paper presents a comprehensive survey of vario…
Image CaptioningLanguage ModelingLanguage ModellingLarge Language Model+6