SALI: Short-term Alignment and Long-term Interaction Network for Colonoscopy Video Polyp Segmentation

2024-06-19 · Qiang Hu, Zhenyu Yi, Ying Zhou, Fang Peng, Mei Liu, Qiang Li, Zhiwei Wang

Colonoscopy videos provide richer information in polyp segmentation for rectal cancer diagnosis. However, the endoscope's fast moving and close-up observing make the current methods suffer from large spatial incoherence and continuous low-quality frames, and thus yield limited segmentation accuracy. In this context, we focus on robust video polyp segmentation by enhancing the adjacent feature consistency and rebuilding the reliable polyp representation. To achieve this goal, we in this paper propose SALI network, a hybrid of Short-term Alignment Module (SAM) and Long-term Interaction Module (LIM). The SAM learns spatial-aligned features of adjacent frames via deformable convolution and further harmonizes them to capture more stable short-term polyp representation. In case of low-quality frames, the LIM stores the historical polyp representations as a long-term memory bank, and explores the retrospective relations to interactively rebuild more reliable polyp features for the current segmentation. Combing SAM and LIM, the SALI network of video segmentation shows a great robustness to the spatial variations and low-visual cues. Benchmark on the large-scale SUNSEG verifies the superiority of SALI over the current state-of-the-arts by improving Dice by 2.1%, 2.5%, 4.1% and 1.9%, for the four test sub-sets, respectively. Codes are at https://github.com/Scatteredrain/SALI.

📄 PDF Abstract BibTeX arXiv:2406.13532

Code (1)

Scatteredrain/SALI 공식 구현 pytorch

Tasks

Segmentation Video Polyp Segmentation Video Segmentation Video Semantic Segmentation

Methods 이 논문이 사용한 방법론

Focus 설명 없음

Deformable Convolution Deformable convolutions add 2D offsets to the regular grid sampling locations in the standard convolution. It enables free…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

SAM 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Transformer A Transformer is a model architecture that eschews recurrence and instead relies entirely on an [attention…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Multi-granularity Correspondence Learning from Long-term Noisy Videos

2024-01-30 · Yijie Lin, Jie Zhang, Zhenyu Huang, Jia Liu 외

Existing video-language studies mainly focus on learning short video clips, leaving long-term temporal dependencies rarely explored due to over-high computational cost of modeling long videos. To address this issue, one …

Action Segmentation Long Video Retrieval (Background Removed)Video Retrieval Video Understanding

A Novel Long-term Iterative Mining Scheme for Video Salient Object Detection

2022-06-20 · Chenglizhao Chen, Hengsen Wang, Yuming Fang, Chong Peng

The existing state-of-the-art (SOTA) video salient object detection (VSOD) models have widely followed short-term methodology, which dynamically determines the balance between spatial and temporal saliency fusion by sole…

Object object-detection Object Detection Salient Object Detection+1

Long-Short Alignment for Effective Long-Context Modeling in LLMs

2025-06-13 · Tianqi Du, Haotian Huang, Yifei Wang, Yisen Wang

Large language models (LLMs) have exhibited impressive performance and surprising emergent properties. However, their effectiveness remains limited by the fixed context window of the transformer architecture, posing chal…

Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learning

2021-06-01 · Bogdan Mazoure, Paul Mineiro, Pavithra Srinath, Reza Sharifi Sedeh 외

We study session-based recommendation scenarios where we want to recommend items to users during sequential interactions to improve their long-term utility. Optimizing a long-term metric is challenging because the learni…

Offline RL Recommendation Systems reinforcement-learning Reinforcement Learning (RL)+1

TSI: Temporal Saliency Integration for Video Action Recognition

2021-06-02 · Haisheng Su, Jinyuan Feng, Dongliang Wang, Weihao Gan 외

Efficient spatiotemporal modeling is an important yet challenging problem for video action recognition. Existing state-of-the-art methods exploit motion clues to assist in short-term temporal modeling through temporal di…

Action Recognition Temporal Action Localization