Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videos
Purpose: Real-time surgical tool tracking is a core component of the future intelligent operating room (OR), because it is highly instrumental to analyze and understand the surgical activities. Current methods for surgical tool tracking in videos need to be trained on data in which the spatial positions of the tools are manually annotated. Generating such training data is difficult and time-consuming. Instead, we propose to use solely binary presence annotations to train a tool tracker for laparoscopic videos. Methods: The proposed approach is composed of a CNN + Convolutional LSTM (ConvLSTM) neural network trained end-to-end, but weakly supervised on tool binary presence labels only. We use the ConvLSTM to model the temporal dependencies in the motion of the surgical tools and leverage its spatio-temporal ability to smooth the class peak activations in the localization heat maps (Lh-maps). Results: We build a baseline tracker on top of the CNN model and demonstrate that our approach based on the ConvLSTM outperforms the baseline in tool presence detection, spatial localization, and motion tracking by over 5.0%, 13.9%, and 12.6%, respectively. Conclusions: In this paper, we demonstrate that binary presence labels are sufficient for training a deep learning tracking model using our proposed method. We also show that the ConvLSTM can leverage the spatio-temporal coherence of consecutive image frames across a surgical video to improve tool presence detection, spatial localization, and motion tracking. keywords: Surgical workflow analysis, tool tracking, weak supervision, spatio-temporal coherence, ConvLSTM, endoscopic videos
Code (1)
Tasks
Instrument RecognitionObject DetectionSurgical tool detectionVideo Object TrackingWeakly-Supervised Object LocalizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Weakly-Supervised Learning for Tool Localization in Laparoscopic Videos
Surgical tool localization is an essential task for the automatic analysis of endoscopic videos. In the literature, existing methods for tool localization, tracking and segmentation require training data that is fully an…
Surgical tool detectionWeakly-supervised LearningSpatially Supervised Recurrent Convolutional Neural Networks for Visual Object Tracking
In this paper, we develop a new approach of spatially supervised recurrent convolutional neural networks for visual object tracking. Our recurrent convolutional network exploits the history of locations as well as the di…
Binary Classificationobject-detectionObject DetectionObject Tracking+2Weakly Supervised Multi-Object Tracking and Segmentation
We introduce the problem of weakly supervised Multi-Object Tracking and Segmentation, i.e. joint weakly supervised instance segmentation and multi-object tracking, in which we do not provide any kind of mask annotation. …
Instance SegmentationMulti-Object TrackingMulti-Object Tracking and SegmentationMulti-Task Learning+6Weakly-Supervised Cell Tracking via Backward-and-Forward Propagation
We propose a weakly-supervised cell tracking method that can train a convolutional neural network (CNN) by using only the annotation of "cell detection" (i.e., the coordinates of cell positions) without association infor…
Cell DetectionCell TrackingWeakly-Supervised Learning via Multi-Lateral Decoder Branching for Tool Segmentation in Robot-Assisted Cardiovascular Catheterization
Robot-assisted catheterization has garnered a good attention for its potentials in treating cardiovascular diseases. However, advancing surgeon-robot collaboration still requires further research, particularly on task-sp…
DecoderSegmentationWeakly-supervised Learning