An Efficient COarse-to-fiNE Alignment Framework @ Ego4D Natural Language Queries Challenge 2022
This technical report describes the CONE approach for Ego4D Natural Language Queries (NLQ) Challenge in ECCV 2022. We leverage our model CONE, an efficient window-centric COarse-to-fiNE alignment framework. Specifically, CONE dynamically slices the long video into candidate windows via a sliding window approach. Centering at windows, CONE (1) learns the inter-window (coarse-grained) semantic variance through contrastive learning and speeds up inference by pre-filtering the candidate windows relevant to the NL query, and (2) conducts intra-window (fine-grained) candidate moments ranking utilizing the powerful multi-modal alignment ability of the contrastive vision-text pre-trained model EgoVLP. On the blind test set, CONE achieves 15.26 and 9.24 for R1@IoU=0.3 and R1@IoU=0.5, respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningNatural Language QueriesMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding
This paper tackles an emerging and challenging problem of long video temporal grounding~(VTG) that localizes video moments related to a natural language (NL) query. Compared with short videos, long videos are also highly…
Contrastive LearningVideo GroundingCAPER: Coarsen, Align, Project, Refine - A General Multilevel Framework for Network Alignment
Network alignment, or the task of finding corresponding nodes in different networks, is an important problem formulation in many application domains. We propose CAPER, a multilevel alignment framework that Coarsens the i…
Riemannian and Symplectic Geometry for Hierarchical Text-Driven Place Recognition
Text-to-point-cloud localization enables robots to understand spatial positions through natural language descriptions, which is crucial for human-robot collaboration in applications such as autonomous driving and last-mi…
Cross-Modal RetrievalAutonomous DrivingPoint CloudsAutomatic database description generation for Text-to-SQL
In the context of the Text-to-SQL task, table and column descriptions are crucial for bridging the gap between natural language and database schema. This report proposes a method for automatically generating effective da…
Text to SQLText-To-SQLCoarse-to-Fine Covid-19 Segmentation via Vision-Language Alignment
Segmentation of COVID-19 lesions can assist physicians in better diagnosis and treatment of COVID-19. However, there are few relevant studies due to the lack of detailed information and high-quality annotation in the COV…
Medical Image SegmentationSegmentation