Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations
We propose Corder, a self-supervised contrastive learning framework for source code model. Corder is designed to alleviate the need of labeled data for code retrieval and code summarization tasks. The pre-trained model of Corder can be used in two ways: (1) it can produce vector representation of code which can be applied to code retrieval tasks that do not have labeled data; (2) it can be used in a fine-tuning process for tasks that might still require label data such as code summarization. The key innovation is that we train the source code model by asking it to recognize similar and dissimilar code snippets through a contrastive learning objective. To do so, we use a set of semantic-preserving transformation operators to generate code snippets that are syntactically diverse but semantically equivalent. Through extensive experiments, we have shown that the code models pretrained by Corder substantially outperform the other baselines for code-to-code retrieval, text-to-code retrieval, and code-to-text summarization tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringCode SummarizationContrastive LearningInformation RetrievalLanguage ModellingMethod name predictionRetrievalSelf-Supervised LearningSource Code SummarizationText SummarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization
In this paper, we present a denoising sequence-to-sequence (seq2seq) autoencoder via contrastive learning for abstractive text summarization. Our model adopts a standard Transformer-based architecture with a multi-layer …
Abstractive Text SummarizationContrastive LearningDecoderDenoising+2Self-Supervised Consistent Quantization for Fully Unsupervised Image Retrieval
Unsupervised image retrieval aims to learn an efficient retrieval system without expensive data annotations, but most existing methods rely heavily on handcrafted feature descriptors or pre-trained feature extractors. To…
Contrastive LearningImage RetrievalQuantizationRetrievalAddressing Leakage in Self-Supervised Contextualized Code Retrieval
We address contextualized code retrieval, the search for code snippets helpful to fill gaps in a partial input program. Our approach facilitates a large-scale self-supervised contrastive training by splitting source code…
Defect DetectionRetrievalContrastive Audio-Visual Masked Autoencoder
In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by co…
Audio ClassificationAudio TaggingContrastive LearningMulti-modal Classification+4Contrastive Learning under Noisy Temporal Self-Supervision for Colonoscopy Videos
Learning robust representations of polyp tracklets is key to enabling multiple AI-assisted colonoscopy applications, from polyp characterization to automated reporting and retrieval. Supervised contrastive learning is an…
Contrastive Learning