Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentation
Self-supervised learning (SSL) approaches have achieved great success when the amount of labeled data is limited. Within SSL, models learn robust feature representations by solving pretext tasks. One such pretext task is contrastive learning, which involves forming pairs of similar and dissimilar input samples, guiding the model to distinguish between them. In this work, we investigate the application of contrastive learning to the domain of medical image analysis. Our findings reveal that MoCo v2, a state-of-the-art contrastive learning method, encounters dimensional collapse when applied to medical images. This is attributed to the high degree of inter-image similarity shared between the medical images. To address this, we propose two key contributions: local feature learning and feature decorrelation. Local feature learning improves the ability of the model to focus on the local regions of the image, while feature decorrelation removes the linear dependence among the features. Our experimental findings demonstrate that our contributions significantly enhance the model's performance in the downstream task of medical segmentation, both in the linear evaluation and full fine-tuning settings. This work illustrates the importance of effectively adapting SSL techniques to the characteristics of medical imaging tasks. The source code will be made publicly available at: https://github.com/CAMMA-public/med-moco
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningImage SegmentationLinear evaluationMedical Image AnalysisMedical Image SegmentationSelf-Supervised LearningSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Understanding Dimensional Collapse in Contrastive Self-supervised Learning
Self-supervised visual representation learning aims to learn useful representations without relying on human annotations. Joint embedding approach bases on maximizing the agreement between embedding vectors from differen…
Contrastive LearningLearning TheoryRepresentation LearningSelf-Supervised LearningSemi-Supervised Contrastive Learning with Orthonormal Prototypes
Contrastive learning has emerged as a powerful method in deep learning, excelling at learning effective representations through contrasting samples from different distributions. However, dimensional collapse, where embed…
Contrastive LearningImage ClassificationObject DetectionDirect Coloring for Self-Supervised Enhanced Feature Decoupling
The success of self-supervised learning (SSL) has been the focus of multiple recent theoretical and empirical studies, including the role of data augmentation (in feature decoupling) as well as complete and dimensional r…
Data AugmentationRepresentation LearningSelf-Supervised LearningPreventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)
Contrastive learning has emerged as a powerful method in deep learning, excelling at learning effective representations through contrasting samples from different distributions. However, neural collapse, where embeddings…
Contrastive LearningWERank: Towards Rank Degradation Prevention for Self-Supervised Learning Using Weight Regularization
A common phenomena confining the representation quality in Self-Supervised Learning (SSL) is dimensional collapse (also known as rank degeneration), where the learned representations are mapped to a low dimensional subsp…
Data AugmentationSelf-Supervised Learning