Sim-DETR: Unlock DETR for Temporal Sentence Grounding
Temporal sentence grounding aims to identify exact moments in a video that correspond to a given textual query, typically addressed with detection transformer (DETR) solutions. However, we find that typical strategies designed to enhance DETR do not improve, and may even degrade, its performance in this task. We systematically analyze and identify the root causes of this abnormal behavior: (1) conflicts between queries from similar target moments and (2) internal query conflicts due to the tension between global semantics and local localization. Building on these insights, we propose a simple yet powerful baseline, Sim-DETR, which extends the standard DETR with two minor modifications in the decoder layers: (1) constraining self-attention between queries based on their semantic and positional overlap and (2) adding query-to-frame alignment to bridge the global and local contexts. Experiments demonstrate that Sim-DETR unlocks the full potential of DETR for temporal sentence grounding, offering a strong baseline for future research.
Code (0)
등록된 구현이 없습니다.
Tasks
Temporal Sentence GroundingSimilar Papers 제목 키워드 기반
Correlation-Guided Query-Dependency Calibration for Video Temporal Grounding
Temporal Grounding is to identify specific moments or highlights from a video corresponding to textual descriptions. Typical approaches in temporal grounding treat all video clips equally during the encoding process rega…
Highlight DetectionMoment RetrievalNatural Language Moment RetrievalRepresentation Learning+1BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos
Temporal sentence grounding aims to localize moments relevant to a language description. Recently, DETR-like approaches achieved notable progress by predicting the center and length of a target moment. However, they suff…
Moment RetrievalNatural Language Moment RetrievalSentenceTemporal Sentence GroundingTubeDETR: Spatio-Temporal Video Grounding with Transformers
We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query. This is a challenging task that requires the joint and efficient modeling of temporal, spatial and multi-modal …
DecoderLanguage-Based Temporal LocalizationNatural Language Visual Groundingobject-detection+6Augmented 2D-TAN: A Two-stage Approach for Human-centric Spatio-Temporal Video Grounding
We propose an effective two-stage approach to tackle the problem of language-based Human-centric Spatio-Temporal Video Grounding (HC-STVG) task. In the first stage, we propose an Augmented 2D Temporal Adjacent Network (A…
Spatio-Temporal Video GroundingVideo GroundingDiversifying Query: Region-Guided Transformer for Temporal Sentence Grounding
Temporal sentence grounding is a challenging task that aims to localize the moment spans relevant to a language description. Although recent DETR-based models have achieved notable progress by leveraging multiple learnab…
AttributeMoment QueriesSentenceTemporal Sentence Grounding