AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation

2026-03-24 · Woojeong Jin, Jaeho Lee, Heeseong Shin, Seungho Jang, Junhwan Heo, Seungryong Kim arxiv

Referring Video Object Segmentation (RVOS) aims to segment a target object throughout a video given a natural language query. Training-free methods for this task follow a common pipeline: a MLLM selects keyframes, grounds the referred object within those frames, and a video segmentation model propagates the results. While intuitive, this design asks the MLLM to make temporal decisions before any object-level evidence is available, limiting both reasoning quality and spatio-temporal coverage. To overcome this, we propose AgentRVOS, a training-free agentic pipeline built on the complementary strengths of SAM3 and a MLLM. Given a concept derived from the query, SAM3 provides reliable perception over the full spatio-temporal extent through generated mask tracks. The MLLM then identifies the target through query-grounded reasoning over this object-level evidence, iteratively pruning guided by SAM3's temporal existence information. Extensive experiments show that AgentRVOS achieves state-of-the-art performance among training-free methods across multiple benchmarks, with consistent results across diverse MLLM backbones. Our project page is available at: https://cvlab-kaist.github.io/AgentRVOS/.

📄 PDF Abstract BibTeX arXiv:2603.23489

Code (0)

등록된 구현이 없습니다.

Tasks

Referring Video Object Segmentation Video Segmentation

AgentRVOS for MeViS-Text Track of 5th PVUW Challenge: 3rd Method

2026-04-20 · Deshui Miao, Chao Yang, Chao Tian, Guoqing Zhu 외 arxiv

This report describes a Ref-VOS pipeline centered on Sa2VA and organized with explicit agent roles. The key idea is that Sa2VA should provide the first dense semantic hypothesis, while an agent loop decides whether that …

Tracking and Understanding Object Transformations

2025-11-06 · Yihong Sun, Xinyu Yang, Jennifer J. Sun, Bharath Hariharan arxiv

Real-world objects frequently undergo state transformations. From an apple being cut into pieces to a butterfly emerging from its cocoon, tracking through these changes is important for understanding real-world objects a…

STRIDE-QA: Visual Question Answering Dataset for Spatiotemporal Reasoning in Urban Driving Scenes

2025-08-14 · Keishi Ishihara, Kento Sasaki, Tsubasa Takahashi, Daiki Shiono 외 arxiv

Vision-Language Models (VLMs) have been applied to autonomous driving to support decision-making in complex real-world scenarios. However, their training on static, web-sourced image-text pairs fundamentally limits the p…

Visual Question Answering Autonomous Driving

Feature Tracks are not Zero-Mean Gaussian

2023-03-25 · Stephanie Tsuei, Wenjie Mo, Stefano Soatto

In state estimation algorithms that use feature tracks as input, it is customary to assume that the errors in feature track positions are zero-mean Gaussian. Using a combination of calibrated camera intrinsics, ground-tr…

State Estimation

Seg2Track++: Probabilistic Track Validation and Data Association for Multi-Object Tracking and Segmentation

2026-06-02 · Diogo Mendonça, Tiago Barros, Cristiano Premebida, Urbano J. Nunes arxiv

Autonomous systems require robust Multi-Object Tracking and Segmentation (MOTS) to operate reliably in dynamic environments, ensuring consistent object identities and precise mask-level delineation. Foundation models suc…

Multi-Object Tracking and Segmentation Zero-shot Generalization Instance Segmentation