Thoughts on Objectives of Sparse and Hierarchical Masked Image Model
Masked image modeling is one of the most poplular objectives of training. Recently, the SparK model has been proposed with superior performance among self-supervised learning models. This paper proposes a new mask pattern for this SparK model, proposing it as the Mesh Mask-ed SparK model. We report the effect of the mask pattern used for image masking in pre-training on performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningSimilar Papers 제목 키워드 기반
Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling
We identify and overcome two key obstacles in extending the success of BERT-style pre-training, or the masked image modeling, to convolutional networks (convnets): (i) convolution operation cannot handle irregular, rando…
2D Object DetectionContrastive LearningDecoderImage Classification+5OptiTree: Hierarchical Thoughts Generation with Tree Search for LLM Optimization Modeling
Optimization modeling is one of the most crucial but technical parts of operations research (OR). To automate the modeling process, existing works have leveraged large language models (LLMs), prompting them to break down…
Green Hierarchical Vision Transformer for Masked Image Modeling
We present an efficient approach for Masked Image Modeling (MIM) with hierarchical Vision Transformers (ViTs), allowing the hierarchical ViTs to discard masked patches and operate only on the visible ones. Our approach c…
GPUObject DetectionRECTOR: Masked Region-Channel-Temporal Modeling for Affective and Cognitive Representation Learning
Affective and cognitive disorders manifest as distributed, time-varying brain network dynamics across regions, channels, and time, challenging robust representation learning from EEG/sEEG for clinical diagnosis. We propo…
EEG Emotion RecognitionRepresentation LearningFrom Neuronal Packets to Thoughtseeds: A Hierarchical Model of Embodied Cognition in the Global Workspace
The emergence of cognition requires a framework that bridges evolutionary principles with neurocomputational mechanisms. This paper introduces the novel "thoughtseed" framework, proposing that cognition arises from the d…
Decision Making