Static Segmentation by Tracking: A Frustratingly Label-Efficient Approach to Fine-Grained Segmentation
We study image segmentation in the biological domain, particularly trait and part segmentation from specimen images (e.g., butterfly wing stripes or beetle body parts). This is a crucial, fine-grained task that aids in understanding the biology of organisms. The conventional approach involves hand-labeling masks, often for hundreds of images per species, and training a segmentation model to generalize these labels to other images, which can be exceedingly laborious. We present a label-efficient method named Static Segmentation by Tracking (SST). SST is built upon the insight: while specimens of the same species have inherent variations, the traits and parts we aim to segment show up consistently. This motivates us to concatenate specimen images into a `pseudo-video'' and reframe trait and part segmentation as a tracking problem. Concretely, SST generates masks for unlabeled images by propagating annotated or predicted masks from the `pseudo-preceding'' images. Powered by Segment Anything Model 2 (SAM~2) initially developed for video segmentation, we show that SST can achieve high-quality trait and part segmentation with merely one labeled image per species -- a breakthrough for analyzing specimen images. We further develop a cycle-consistent loss to fine-tune the model, again using one labeled image. Additionally, we highlight the broader potential of SST, including one-shot instance segmentation on images taken in the wild and trait-based image retrieval.
Code (0)
등록된 구현이 없습니다.
Tasks
Image RetrievalImage SegmentationInstance SegmentationOne-Shot Instance SegmentationSegmentationSemantic SegmentationVideo SegmentationVideo Semantic SegmentationSimilar Papers 제목 키워드 기반
FoxInst: A Frustratingly Simple Baseline for Weakly Few-shot Instance Segmentation
We propose the first weakly-supervised few-shot instance segmentation task and a frustratingly simple but strong baseline model, FoxInst. Our work is distinguished from other approaches in that our method is trained with…
Few-shot Instance SegmentationInstance SegmentationSemantic SegmentationZero-Shot Open-Vocabulary Tracking with Large Pre-Trained Models
Object tracking is central to robot perception and scene understanding. Tracking-by-detection has long been a dominant paradigm for object tracking of specific object categories. Recently, large-scale pre-trained models …
ObjectObject TrackingOptical Flow EstimationScene Understanding+3JOTS: Joint Online Tracking and Segmentation
We present a novel Joint Online Tracking and Segmentation (JOTS) algorithm which integrates the multi-part tracking and segmentation into a unified energy optimization framework to handle the video segmentation task. The…
SegmentationVideo SegmentationVideo Semantic SegmentationUnsupervised Fish Trajectory Tracking and Segmentation
DNN for fish tracking and segmentation based on high-quality labels is expensive. Alternative unsupervised approaches rely on spatial and temporal variations that naturally occur in video data to generate noisy pseudo-gr…
Optical Flow EstimationSegmentationReturn of Frustratingly Easy Unsupervised Video Domain Adaptation
Unsupervised video domain adaptation (UVDA) is a practical but under-explored problem. In this paper, we propose a frustratingly easy UVDA method, called MetaTrans. Specifically, MetaTrans adopts a concise learning objec…
Action RecognitionDomain Adaptation