Papers TGIF-Frame
“TGIF-Frame” 태그가 달린 논문 15편 · 필터 해제
Open-vocabulary Video Question Answering: A New Benchmark for Evaluating the Generalizability of Video Question Answering Models
Video Question Answering (VideoQA) is a challenging task that entails complex multi-modal reasoning. In contrast to multiple-choice VideoQA which aims to predict the answer given several options, the goal of open-ended V…
Multiple-choiceQuestion AnsweringTGIF-FrameVideo Question Answering+2Self-Adaptive Sampling for Efficient Video Question-Answering on Image--Text Models
Video question-answering is a fundamental task in the field of video understanding. Although current vision--language models (VLMs) equipped with Video Transformers have enabled temporal modeling and yielded superior res…
Question AnsweringTGIF-FrameVideo Question AnsweringVideo Understanding+2Lightweight Recurrent Cross-modal Encoder for Video Question Answering
A video question answering task essentially boils down to how to fuse the information between text and video effectively to predict an answer. Most works employ a transformer encoder as a cross-modal encoder to fuse both…
Action RecognitionQuestion AnsweringTGIF-ActionTGIF-Frame+3COSA: Concatenated Sample Pretrained Vision-Language Foundation Model
Due to the limited scale and quality of video-text training corpus, most vision-language foundation models employ image-text datasets for pretraining and primarily focus on modeling visually semantic representations whil…
FormmodelQuestion AnsweringRetrieval+6VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to es…
Audio captioningAudio-Visual CaptioningAudio-visual Question AnsweringCross-Modal Retrieval+12VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blending
Large-scale image-text contrastive pre-training models, such as CLIP, have been demonstrated to effectively learn high-quality multimodal representations. However, there is limited research on learning video-text represe…
Question AnsweringRetrievalText RetrievalTGIF-Frame+5VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly mo…
Audio captioningAudio-Video Question Answering (AVQA)Audio-Visual CaptioningAudio-visual Question Answering+16MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Models
Foundation models have shown outstanding performance and generalization capabilities across domains. Since most studies on foundation models mainly focus on the pretraining phase, a naive strategy to minimize a single ta…
Auxiliary LearningMultimodal Sentiment AnalysisQuestion AnsweringRetrieval+9MuLTI: Efficient Video-and-Language Understanding with Text-Guided MultiWay-Sampler and Multiple Choice Modeling
Video-and-language understanding has a variety of applications in the industry, such as video question answering, text-video retrieval, and multi-label classification. Existing video-and-language understanding methods ge…
Multi-Label ClassificationMUlTI-LABEL-ClASSIFICATIONMultiple-choiceQuestion Answering+7mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video
Recent years have witnessed a big convergence of language, vision, and multi-modal pretraining. In this work, we present mPLUG-2, a new unified paradigm with modularized design for multi-modal pretraining, which can bene…
Action ClassificationImage ClassificationImage RetrievalTGIF-Frame+7HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training
Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-languag…
cross-modal alignmentTGIF-ActionTGIF-FrameTGIF-Transition+6An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling
Masked visual modeling (MVM) has been recently proven effective for visual pre-training. While similar reconstructive objectives on video inputs (e.g., masked frame modeling) have been explored in video-language (VidL) p…
Fill MaskOptical Flow EstimationQuestion AnsweringRetrieval+8Clover: Towards A Unified Video-Language Alignment and Fusion Model
Building a universal Video-Language model for solving various video understanding tasks (\emph{e.g.}, text-video retrieval, video question answering) is an open challenge to the machine learning field. Towards this goal,…
Language ModelingLanguage ModellingQuestion AnsweringRetrieval+9Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Video question answering (VideoQA) is a complex task that requires diverse multi-modal data for training. Manual annotation of question and answers for videos, however, is tedious and prohibits scalability. To tackle thi…
Fill MaskLanguage ModelingLanguage ModellingMasked Language Modeling+8All in One: Exploring Unified Video-Language Pre-training
Mainstream Video-Language Pre-training models \cite{actbert,clipbert,violet} consist of three parts, a video encoder, a text encoder, and a video-text fusion Transformer. They pursue better performance via utilizing heav…
AllLanguage ModellingMultiple-choiceQuestion Answering+9