The Effectiveness of Pre-Trained Code Embeddings
Word embeddings are widely used in machine learning based natural language processing systems. It is common to use pre-trained word embeddings which provide benefits such as reduced training time and improved overall performance. There has been a recent interest in applying natural language processing techniques to programming languages. However, none of this recent work uses pre-trained embeddings on code tokens. Using extreme summarization as the downstream task, we show that using pre-trained embeddings on code tokens provides the same benefits as it does to natural languages, achieving: over 1.9x speedup, 5\% improvement in test loss, 4\% improvement in F1 scores, and resistance to over-fitting. We also show that the choice of language used for the embeddings does not have to match that of the task to achieve these benefits and that even embeddings pre-trained on human languages provide these benefits to programming languages.
Code (0)
등록된 구현이 없습니다.
Tasks
Extreme SummarizationWord EmbeddingsSimilar Papers 제목 키워드 기반
CS-Embed at SemEval-2020 Task 9: The effectiveness of code-switched word embeddings for sentiment analysis
The growing popularity and applications of sentiment analysis of social media posts has naturally led to sentiment analysis of posts written in multiple languages, a practice known as code-switching. While recent researc…
Multilingual Word EmbeddingsSentiment AnalysisWord EmbeddingsReconsidering Degeneration of Token Embeddings with Definitions for Encoder-based Pre-trained Language Models
Learning token embeddings based on token co-occurrence statistics has proven effective for both pre-training and fine-tuning in natural language processing. However, recent studies have pointed out that the distribution …
Text SummarizationDimension Reduction for Efficient Dense Retrieval via Conditional Autoencoder
Dense retrievers encode queries and documents and map them in an embedding space using pre-trained language models. These embeddings need to be high-dimensional to fit training signals and guarantee the retrieval effecti…
Dimensionality ReductionInformation RetrievalRetrievalIsotropy Matters: Soft-ZCA Whitening of Embeddings for Semantic Code Search
Low isotropy in an embedding space impairs performance on tasks involving semantic inference. Our study investigates the impact of isotropy on semantic code search performance and explores post-processing techniques to m…
Code SearchHashing-Baseline: Rethinking Hashing in the Age of Pretrained Models
Information retrieval with compact binary embeddings, also referred to as hashing, is crucial for scalable fast search applications, yet state-of-the-art hashing methods require expensive, scenario-specific training. In …
Information RetrievalImage Retrieval