SAMSNeRF: Segment Anything Model (SAM) Guides Dynamic Surgical Scene Reconstruction by Neural Radiance Field (NeRF)
The accurate reconstruction of surgical scenes from surgical videos is critical for various applications, including intraoperative navigation and image-guided robotic surgery automation. However, previous approaches, mainly relying on depth estimation, have limited effectiveness in reconstructing surgical scenes with moving surgical tools. To address this limitation and provide accurate 3D position prediction for surgical tools in all frames, we propose a novel approach called SAMSNeRF that combines Segment Anything Model (SAM) and Neural Radiance Field (NeRF) techniques. Our approach generates accurate segmentation masks of surgical tools using SAM, which guides the refinement of the dynamic surgical scene reconstruction by NeRF. Our experimental results on public endoscopy surgical videos demonstrate that our approach successfully reconstructs high-fidelity dynamic surgical scenes and accurately reflects the spatial information of surgical tools. Our proposed approach can significantly enhance surgical navigation and automation by providing surgeons with accurate 3D position information of surgical tools during surgery.The source code will be released soon.
Code (0)
등록된 구현이 없습니다.
Tasks
Depth EstimationNeRFPositionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning
Surgical video segmentation is a critical task in computer-assisted surgery and is vital for enhancing surgical quality and patient outcomes. Recently, the Segment Anything Model 2 (SAM2) framework has shown superior adv…
SegmentationVideo SegmentationVideo Semantic SegmentationZero-Shot Surgical Tool Segmentation in Monocular Video Using Segment Anything Model 2
The Segment Anything Model 2 (SAM 2) is the latest generation foundation model for image and video segmentation. Trained on the expansive Segment Anything Video (SA-V) dataset, which comprises 35.5 million masks across 5…
DiversitySegmentationVideo SegmentationVideo Semantic Segmentation+1Dynamically evolving segment anything model with continuous learning for medical image segmentation
Medical image segmentation is essential for clinical diagnosis, surgical planning, and treatment monitoring. Traditional approaches typically strive to tackle all medical image segmentation scenarios via one-time learnin…
DiversityImage SegmentationMedical Image SegmentationMRI segmentation+2ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Tracking
Surgical scene segmentation is critical in computer-assisted surgery and is vital for enhancing surgical quality and patient outcomes. Recently, referring surgical segmentation is emerging, given its advantage of providi…
DiversityMambaScene SegmentationSegmentationIs Segment Anything Model 2 All You Need for Surgery Video Segmentation? A Systematic Evaluation
Surgery video segmentation is an important topic in the surgical AI field. It allows the AI model to understand the spatial information of a surgical scene. Meanwhile, due to the lack of annotated surgical data, surgery …
AllSegmentationVideo SegmentationVideo Semantic Segmentation