An integrated language-vision foundation model for conversational diagnostics and triaging in primary eye care
Current deep learning models are mostly task specific and lack a user-friendly interface to operate. We present Meta-EyeFM, a multi-function foundation model that integrates a large language model (LLM) with vision foundation models (VFMs) for ocular disease assessment. Meta-EyeFM leverages a routing mechanism to enable accurate task-specific analysis based on text queries. Using Low Rank Adaptation, we fine-tuned our VFMs to detect ocular and systemic diseases, differentiate ocular disease severity, and identify common ocular signs. The model achieved 100% accuracy in routing fundus images to appropriate VFMs, which achieved $\ge$ 82.2% accuracy in disease detection, $\ge$ 89% in severity differentiation, $\ge$ 76% in sign identification. Meta-EyeFM was 11% to 43% more accurate than Gemini-1.5-flash and ChatGPT-4o LMMs in detecting various eye diseases and comparable to an ophthalmologist. This system offers enhanced usability and diagnostic performance, making it a valuable decision support tool for primary eye care or an online LLM for fundus evaluation.
Code (0)
등록된 구현이 없습니다.
Tasks
DiagnosticLanguage ModelingLanguage ModellingLarge Language ModelSimilar Papers 제목 키워드 기반
Insect-Foundation: A Foundation Model and Large Multimodal Dataset for Vision-Language Insect Understanding
Multimodal conversational generative AI has shown impressive capabilities in various vision and language understanding through learning massive text-image data. However, current conversational models still lack knowledge…
General KnowledgeQuestion AnsweringSelf-Supervised LearningSafety at Scale: A Comprehensive Survey of Large Model Safety
The rapid advancement of large models, driven by their exceptional abilities in learning and generalization through large-scale pre-training, has reshaped the landscape of Artificial Intelligence (AI). These models are n…
Autonomous DrivingData PoisoningModel extractionRecommendation Systems+2Large AI Models in Dental Healthcare: From General-Purpose Systems to Domain-Specific Foundation Models
Background: Oral diseases affect nearly 3.5 billion people worldwide, yet the comparative clinical potential of large-scale AI models in dentistry remains poorly understood. Three distinct model categories have emerged: …
VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledge
The need for improved diagnostic methods in ophthalmology is acute, especially in the less developed regions with limited access to specialists and advanced equipment. Therefore, we introduce VisionUnite, a novel vision-…
Clinical KnowledgeDiagnosticAn Impedance Sensing Platform for Monitoring Heterogeneous Connectivity and Diagnostics in Lab-on-a-Chip Systems
Reliable hardware connectivity is vital in heterogeneous integrated systems. For example, in digital microfluidics lab-on-a-chip systems, there are hundreds of physical connections required between a micro-electro-mechan…