Pixel-Wise Recognition for Holistic Surgical Scene Understanding
This paper presents the Holistic and Multi-Granular Surgical Scene Understanding of Prostatectomies (GraSP) dataset, a curated benchmark that models surgical scene understanding as a hierarchy of complementary tasks with varying levels of granularity. Our approach encompasses long-term tasks, such as surgical phase and step recognition, and short-term tasks, including surgical instrument segmentation and atomic visual actions detection. To exploit our proposed benchmark, we introduce the Transformers for Actions, Phases, Steps, and Instrument Segmentation (TAPIS) model, a general architecture that combines a global video feature extractor with localized region proposals from an instrument segmentation model to tackle the multi-granularity of our benchmark. Through extensive experimentation in ours and alternative benchmarks, we demonstrate TAPIS's versatility and state-of-the-art performance across different tasks. This work represents a foundational step forward in Endoscopic Vision, offering a novel framework for future research towards holistic surgical scene understanding.
Code (3)
Tasks
Scene UnderstandingSegmentationSurgical phase recognitionSimilar Papers 제목 키워드 기반
Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs
Surgical scene understanding is a cornerstone of computer-assisted intervention. While recent advances, particularly in surgical image segmentation, have driven progress, real-world clinical applications require a more h…
Scene UnderstandingImage SegmentationVisual GroundingTowards Holistic Surgical Scene Understanding
Most benchmarks for studying surgical interventions focus on a specific challenge instead of leveraging the intrinsic complementarity among different tasks. In this work, we present a new experimental framework towards h…
Action RecognitionAtomic action recognitionScene UnderstandingSurgical phase recognitionTemporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions
Effective multi-task learning for surgical scene understanding is fundamentally hindered by annotation granularity mismatch; temporal workflow tasks such as phase recognition, step recognition and anticipation benefit fr…
Surgical phase recognitionOptical Flow EstimationRepresentation LearningMulti-Task LearningDynamic Scene Graph Representation for Surgical Video
Surgical videos captured from microscopic or endoscopic imaging devices are rich but complex sources of information, depicting different tools and anatomical structures utilized during an extended amount of time. Despite…
Towards Holistic Surgical Scene Graph
Surgical scene understanding is crucial for computer-assisted intervention systems, requiring visual comprehension of surgical scenes that involves diverse elements such as surgical tools, anatomical structures, and thei…
Action Triplet RecognitionScene Understanding