Learning Video Representations using Contrastive Bidirectional Transformer
This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.
Code (0)
등록된 구현이 없습니다.
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Representation LearningSelf-Supervised Learningspeech-recognitionSpeech RecognitionVideo CaptioningVideo ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Towards Contrastive Learning in Music Video Domain
Contrastive learning is a powerful way of learning multimodal representations across various domains such as image-caption retrieval and audio-visual representation learning. In this work, we investigate if these finding…
Contrastive LearningGenre classificationMusic TaggingRepresentation Learning+1CoBERL: Contrastive BERT for Reinforcement Learning
Many reinforcement learning (RL) agents require a large amount of experience to solve tasks. We propose Contrastive BERT for RL (CoBERL), an agent that combines a new contrastive loss and a hybrid LSTM-transformer archit…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning
Cross-modal correlation provides an inherent supervision for video unsupervised representation learning. Existing methods focus on distinguishing different video clips by visual and audio representations. We human visual…
Contrastive LearningRepresentation LearningSITAR: Semi-supervised Image Transformer for Action Recognition
Recognizing actions from a limited set of labeled videos remains a challenge as annotating visual data is not only tedious but also can be expensive due to classified nature. Moreover, handling spatio-temporal data using…
Action RecognitionContrastive LearningTemporal Action LocalizationTowards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers
Autoregressive transformers have shown remarkable success in video generation. However, the transformers are prohibited from directly learning the long-term dependency in videos due to the quadratic complexity of self-at…
Video Generation