E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model
The development of 3D medical vision-language models holds significant potential for disease diagnosis and patient treatment. However, compared to 2D medical images, 3D medical images, such as CT scans, face challenges related to limited training data and high dimension, which severely restrict the progress of 3D medical vision-language models. To address these issues, we collect a large amount of unlabeled 3D CT data and utilize self-supervised learning to construct a 3D visual foundation model for extracting 3D visual features. Then, we apply 3D spatial convolutions to aggregate and project high-level image features, reducing computational complexity while preserving spatial information. We also construct two instruction-tuning datasets based on BIMCV-R and CT-RATE to fine-tune the 3D vision-language model. Our model demonstrates superior performance compared to existing methods in report generation, visual question answering, and disease diagnosis. Code and data will be made publicly available soon.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingQuestion AnsweringSelf-Supervised LearningVisual Question AnsweringSimilar Papers 제목 키워드 기반
Vision-Language Enhanced Foundation Model for Semi-supervised Medical Image Segmentation
Semi-supervised learning (SSL) has emerged as an effective paradigm for medical image segmentation, reducing the reliance on extensive expert annotations. Meanwhile, vision-language models (VLMs) have demonstrated strong…
Semi-supervised Medical Image SegmentationKnowledge-enhanced Visual-Language Pre-training on Chest Radiology Images
While multi-modal foundation models pre-trained on large-scale data have been successful in natural language understanding and vision recognition, their use in medical domains is still limited due to the fine-grained nat…
Natural Language UnderstandingRepresentation LearningGrounded Knowledge-Enhanced Medical VLP for Chest X-Ray
Medical vision-language pre-training has emerged as a promising approach for learning domain-general representations of medical image and text. Current algorithms that exploit the global and local alignment between medic…
Medical Visual Question AnsweringQuestion AnsweringVisual Question AnsweringLocation-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models
Fine-grained visual representations are essential for medical image analysis, particularly when diagnostically relevant evidence is subtle and spatially localized. Modern transformer-based medical vision encoders must th…
Visual Question AnsweringSelf-Supervised LearningRepresentation LearningPhrase GroundingRescind: Countering Image Misconduct in Biomedical Publications with Vision-Language and State-Space Modeling
Scientific image manipulation in biomedical publications poses a growing threat to research integrity and reproducibility. Unlike natural image forensics, biomedical forgery detection is uniquely challenging due to domai…
Image Manipulation