Zero-Shot Surgical Tool Segmentation in Monocular Video Using Segment Anything Model 2
The Segment Anything Model 2 (SAM 2) is the latest generation foundation model for image and video segmentation. Trained on the expansive Segment Anything Video (SA-V) dataset, which comprises 35.5 million masks across 50.9K videos, SAM 2 advances its predecessor's capabilities by supporting zero-shot segmentation through various prompts (e.g., points, boxes, and masks). Its robust zero-shot performance and efficient memory usage make SAM 2 particularly appealing for surgical tool segmentation in videos, especially given the scarcity of labeled data and the diversity of surgical procedures. In this study, we evaluate the zero-shot video segmentation performance of the SAM 2 model across different types of surgeries, including endoscopy and microscopy. We also assess its performance on videos featuring single and multiple tools of varying lengths to demonstrate SAM 2's applicability and effectiveness in the surgical domain. We found that: 1) SAM 2 demonstrates a strong capability for segmenting various surgical videos; 2) When new tools enter the scene, additional prompts are necessary to maintain segmentation accuracy; and 3) Specific challenges inherent to surgical videos can impact the robustness of SAM 2.
Code (1)
Tasks
DiversitySegmentationVideo SegmentationVideo Semantic SegmentationZero Shot SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SurgPose: Generalisable Surgical Instrument Pose Estimation using Zero-Shot Learning and Stereo Vision
Accurate pose estimation of surgical tools in Robot-assisted Minimally Invasive Surgery (RMIS) is essential for surgical navigation and robot control. While traditional marker-based methods offer accuracy, they face chal…
Depth EstimationInstance SegmentationPose EstimationSemantic Segmentation+1ZEAL: Surgical Skill Assessment with Zero-shot Tool Inference Using Unified Foundation Model
Surgical skill assessment is paramount for ensuring patient safety and enhancing surgical outcomes. This study addresses the need for efficient and objective evaluation methods by introducing ZEAL (surgical skill assessm…
SegmentationMore than Segmentation: Benchmarking SAM 3 for Segmentation, 3D Perception, and Reconstruction in Robotic Surgery
The recent SAM 3 and SAM 3D have introduced significant advancements over the predecessor, SAM 2, particularly with the integration of language-based segmentation and enhanced 3D perception capabilities. SAM 3 supports z…
Monocular Depth EstimationVideo SegmentationIs Segment Anything Model 2 All You Need for Surgery Video Segmentation? A Systematic Evaluation
Surgery video segmentation is an important topic in the surgical AI field. It allows the AI model to understand the spatial information of a surgical scene. Meanwhile, due to the lack of annotated surgical data, surgery …
AllSegmentationVideo SegmentationVideo Semantic SegmentationSurgicalVLM-Agent: Towards an Interactive AI Co-Pilot for Pituitary Surgery
Image-guided surgery demands adaptive, real-time decision support, yet static AI models struggle with structured task planning and providing interactive guidance. Large vision-language models (VLMs) offer a promising sol…
Activity RecognitionAnatomyQuestion AnsweringSegmentation+4