How to Efficiently Adapt Large Segmentation Model(SAM) to Medical Images
The emerging scale segmentation model, Segment Anything (SAM), exhibits impressive capabilities in zero-shot segmentation for natural images. However, when applied to medical images, SAM suffers from noticeable performance drop. To make SAM a real ``foundation model" for the computer vision community, it is critical to find an efficient way to customize SAM for medical image dataset. In this work, we propose to freeze SAM encoder and finetune a lightweight task-specific prediction head, as most of weights in SAM are contributed by the encoder. In addition, SAM is a promptable model, while prompt is not necessarily available in all application cases, and precise prompts for multiple class segmentation are also time-consuming. Therefore, we explore three types of prompt-free prediction heads in this work, include ViT, CNN, and linear layers. For ViT head, we remove the prompt tokens in the mask decoder of SAM, which is named AutoSAM. AutoSAM can also generate masks for different classes with one single inference after modification. To evaluate the label-efficiency of our finetuning method, we compare the results of these three prediction heads on a public medical image segmentation dataset with limited labeled data. Experiments demonstrate that finetuning SAM significantly improves its performance on medical image dataset, even with just one labeled volume. Moreover, AutoSAM and CNN prediction head also has better segmentation accuracy than training from scratch and self-supervised learning approaches when there is a shortage of annotations.
Code (1)
Tasks
Image SegmentationMedical Image SegmentationPredictionSegmentationSelf-Supervised LearningSemantic SegmentationZero Shot SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Tri-Plane Mamba: Efficiently Adapting Segment Anything Model for 3D Medical Images
General networks for 3D medical image segmentation have recently undergone extensive exploration. Behind the exceptional performance of these networks lies a significant demand for a large volume of pixel-level annotated…
Image SegmentationMambaMedical Image SegmentationOrgan Segmentation+2VLSM-Adapter: Finetuning Vision-Language Segmentation Efficiently with Lightweight Blocks
Foundation Vision-Language Models (VLMs) trained using large-scale open-domain images and text pairs have recently been adapted to develop Vision-Language Segmentation Models (VLSMs) that allow providing text prompts dur…
Image SegmentationMedical Image SegmentationSegmentationSemantic Segmentation+1Uncertainty-Aware Adapter: Adapting Segment Anything Model (SAM) for Ambiguous Medical Image Segmentation
The Segment Anything Model (SAM) gained significant success in natural image segmentation, and many methods have tried to fine-tune it to medical image segmentation. An efficient way to do so is by using Adapters, specia…
Image SegmentationMedical Image SegmentationOptic Cup SegmentationSegmentation+1S-SAM: SVD-based Fine-Tuning of Segment Anything Model for Medical Image Segmentation
Medical image segmentation has been traditionally approached by training or fine-tuning the entire model to cater to any new modality or dataset. However, this approach often requires tuning a large number of parameters …
Image SegmentationMedical Image SegmentationSemantic SegmentationAdvancing Medical Image Segmentation with Mini-Net: A Lightweight Solution Tailored for Efficient Segmentation of Medical Images
Accurate segmentation of anatomical structures and abnormalities in medical images is crucial for computer-aided diagnosis and analysis. While deep learning techniques excel at this task, their computational demands pose…
Image SegmentationMedical Image SegmentationSegmentationSemantic Segmentation