Masked Feature Modelling: Feature Masking for the Unsupervised Pre-training of a Graph Attention Network Block for Bottom-up Video Event Recognition
In this paper, we introduce Masked Feature Modelling (MFM), a novel approach for the unsupervised pre-training of a Graph Attention Network (GAT) block. MFM utilizes a pretrained Visual Tokenizer to reconstruct masked features of objects within a video, leveraging the MiniKinetics dataset. We then incorporate the pre-trained GAT block into a state-of-the-art bottom-up supervised video-event recognition architecture, ViGAT, to improve the model's starting point and overall accuracy. Experimental evaluations on the YLI-MED dataset demonstrate the effectiveness of MFM in improving event recognition performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Graph AttentionUnsupervised Pre-trainingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MINR: Implicit Neural Representations with Masked Image Modelling
Self-supervised learning methods like masked autoencoders (MAE) have shown significant promise in learning robust feature representations, particularly in image reconstruction-based pretraining task. However, their perfo…
Self-Supervised LearningImage ReconstructionWord Order Matters when you Increase Masking
Word order, an essential property of natural languages, is injected in Transformer-based neural language models using position encoding. However, recent experiments have shown that explicit position encoding is not alway…
Language ModellingPositionSentenceDual form Complementary Masking for Domain-Adaptive Image Segmentation
Recent works have correlated Masked Image Modeling (MIM) with consistency regularization in Unsupervised Domain Adaptation (UDA). However, they merely treat masking as a special form of deformation on the input images an…
Unsupervised Domain AdaptationRepresentation LearningDomain GeneralizationImage SegmentationInforMask: Unsupervised Informative Masking for Language Model Pretraining
Masked language modeling is widely used for pretraining large language models for natural language understanding (NLU). However, random masking is suboptimal, allocating an equal masking rate for all tokens. In this pape…
Language ModelingLanguage ModellingMasked Language Modelingmodel+2Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text
Masked language modeling has become a widely adopted unsupervised technique to pre-train language models. However, the process of selecting tokens for masking is random, and the percentage of masked tokens is typically f…
Authorship AttributionLanguage ModelingLanguage ModellingMasked Language Modeling+4