Deep Multimodal Learning: An Effective Method for Video Classification
Videos have become ubiquitous on the Internet. And video analysis can provide lots of information for detecting and recognizing objects as well as help people understand human actions and interactions with the real world. However, facing data as huge as TB level, effective methods should be applied. Recurrent neural network (RNN) architecture has wildly been used on many sequential learning problems such as Language Model, Time-Series Analysis, etc. In this paper, we propose some variations of RNN such as stacked bidirectional LSTM/GRU network with attention mechanism to categorize large-scale video data. We also explore different multimodal fusion methods. Our model combines both visual and audio information on both video and frame level and received great result. Ensemble methods are also applied. Because of its multimodal characteristics, we decide to call this method Deep Multimodal Learning(DML). Our DML-based model was trained on Google Cloud and our own server and was tested in a well-known video classification competition on Kaggle held by Google.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationGeneral ClassificationLanguage ModelingLanguage ModellingTime SeriesTime Series AnalysisVideo ClassificationSimilar Papers 제목 키워드 기반
Multimodal Hate Detection Using Dual-Stream Graph Neural Networks
Hateful videos present serious risks to online safety and real-world well-being, necessitating effective detection methods. Although multimodal classification approaches integrating information from several modalities ou…
Video ClassificationGraph Neural NetworkMultimodal Semantic Attention Network for Video Captioning
Inspired by the fact that different modalities in videos carry complementary information, we propose a Multimodal Semantic Attention Network(MSAN), which is a new encoder-decoder framework incorporating multimodal semant…
AttributeDecoderGeneral ClassificationMulti-Label Classification+2Efficient Selective Audio Masked Multimodal Bottleneck Transformer for Audio-Video Classification
Audio and video are two most common modalities in the mainstream media platforms, e.g., YouTube. To learn from multimodal videos effectively, in this work, we propose a novel audio-video recognition approach termed audio…
Action RecognitionContrastive LearningVideo ClassificationVideo RecognitionEnd-to-end Generative Pretraining for Multimodal Video Captioning
Recent video and language pretraining frameworks lack the ability to generate sentences. We present Multimodal Video Generative Pretraining (MV-GPT), a new pretraining framework for learning from unlabelled videos which …
Action ClassificationDecoderRetrievalSentence+3Towards Good Practices for Multi-modal Fusion in Large-scale Video Classification
Leveraging both visual frames and audio has been experimentally proven effective to improve large-scale video classification. Previous research on video classification mainly focuses on the analysis of visual content amo…
ClassificationGeneral ClassificationVideo Classification