Masked Event Modeling: Self-Supervised Pretraining for Event Cameras
Event cameras asynchronously capture brightness changes with low latency, high temporal resolution, and high dynamic range. However, annotation of event data is a costly and laborious process, which limits the use of deep learning methods for classification and other semantic tasks with the event modality. To reduce the dependency on labeled event data, we introduce Masked Event Modeling (MEM), a self-supervised framework for events. Our method pretrains a neural network on unlabeled events, which can originate from any event camera recording. Subsequently, the pretrained model is finetuned on a downstream task, leading to a consistent improvement of the task accuracy. For example, our method reaches state-of-the-art classification accuracy across three datasets, N-ImageNet, N-Cars, and N-Caltech101, increasing the top-1 accuracy of previous work by significant margins. When tested on real-world event data, MEM is even superior to supervised RGB-based pretraining. The models pretrained with MEM are also label-efficient and generalize well to the dense task of semantic image segmentation.
Code (1)
Tasks
ClassificationEvent-based visionImage SegmentationSemantic SegmentationSimilar Papers 제목 키워드 기반
Understanding Self-Supervised Pretraining with Part-Aware Representation Learning
In this paper, we are interested in understanding self-supervised pretraining through studying the capability that self-supervised representation pretraining methods learn part-aware representations. The study is mainly …
Contrastive LearningObjectRepresentation LearningJiTTER: Jigsaw Temporal Transformer for Event Reconstruction for Self-Supervised Sound Event Detection
Sound event detection (SED) has significantly benefited from self-supervised learning (SSL) approaches, particularly masked audio transformer for SED (MAT-SED), which leverages masked block prediction to reconstruct miss…
Boundary DetectionEvent DetectionRepresentation LearningSelf-Supervised Learning+1Selfie: Self-supervised Pretraining for Image Embedding
We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as…
Language ModelingLanguage ModellingMasked Language ModelingAn Empirical Study Of Self-supervised Learning Approaches For Object Detection With Transformers
Self-supervised learning (SSL) methods such as masked language modeling have shown massive performance gains by pretraining transformer models for a variety of natural language processing tasks. The follow-up research ad…
image-classificationImage ClassificationImage ReconstructionLanguage Modeling+6Energy-Inspired Self-Supervised Pretraining for Vision Models
Motivated by the fact that forward and backward passes of a deep network naturally form symmetric mappings between input and output representations, we introduce a simple yet effective self-supervised vision model pretra…
ColorizationDecoderDenoisingImage Restoration+1